https://lakefs.io/ logo
Title
t

Thomas Vander Wal

10/09/2022, 5:20 AM
I’ve been upgrading our lakeFS instances to the newest version but having issues on our main one. Error in thread but the 0.80.0 migration is failing due to a null field issue, which makes me think something may have gotten out of sync inside the database.
time="2022-10-09T05:09:17Z" level=info msg="Migrated 100000 entries" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" created_at="2022-10-09 05:09:03.593783788 +0000 UTC" db_schema_version=1 lakefs_version=0.80.0 package_name=graveler
time="2022-10-09T05:09:27Z" level=info msg="Migrated 200000 entries" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" created_at="2022-10-09 05:09:03.593783788 +0000 UTC" db_schema_version=1 lakefs_version=0.80.0 package_name=graveler
time="2022-10-09T05:09:38Z" level=info msg="Migrated 300000 entries" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" created_at="2022-10-09 05:09:03.593783788 +0000 UTC" db_schema_version=1 lakefs_version=0.80.0 package_name=graveler
time="2022-10-09T05:09:49Z" level=info msg="Migrated 400000 entries" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" created_at="2022-10-09 05:09:03.593783788 +0000 UTC" db_schema_version=1 lakefs_version=0.80.0 package_name=graveler
time="2022-10-09T05:10:00Z" level=info msg="Successfully migrated package to KV" func=pkg/kv/postgres.Migrate.func1 file="build/pkg/kv/postgres/migrate.go:129" pkg_id=graveler
time="2022-10-09T05:10:00Z" level=info msg="graveler Package Migration took 56.506001978s" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" TempDir=/tmp/kv_migrate_4110592530
time="2022-10-09T05:10:00Z" level=info msg="KV Migration took 56.506086254s" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" TempDir=/tmp/kv_migrate_4110592530
Failed to setup DB: 1 error occurred:
   * failed migration on package multiparts: scany: scan row into struct fields: can't scan into dest[5]: cannot scan null into *string
y

Yoni Augarten

10/09/2022, 5:27 AM
Hey @Thomas Vander Wal, can you please specify from and to which version you are upgrading?
t

Thomas Vander Wal

10/09/2022, 5:28 AM
Hey Yoni! I’m currently on 0.66.0 and going to to 0.80.0 for the migration, before upgrading to 0.83.2 This did work for another instance that was on 0.63.0
y

Yoni Augarten

10/09/2022, 5:29 AM
Thank you, I'm checking it out and will get back to you shortly
:heart_lakefs: 1
t

Thomas Vander Wal

10/09/2022, 5:45 AM
It does look like the 0.69.0 migration did succeed so I can confirm my issue is with the latest
y

Yoni Augarten

10/09/2022, 5:53 AM
@Thomas Vander Wal, do you mean a migration from or to 0.69.0?
t

Thomas Vander Wal

10/09/2022, 5:53 AM
To 0.69.0. The only way I was able to get the service back up and running was to switch to that version.
y

Yoni Augarten

10/09/2022, 5:54 AM
I see. I'm trying to reproduce the issue on my end. Will update here
It seems like there is an issue with the migration process, it is related to a database table that keeps track of multipart uploads into lakeFS. @Thomas Vander Wal, to unblock you, I can suggest a query to run directly on the database before upgrading. Is that something you can do?
t

Thomas Vander Wal

10/09/2022, 6:13 AM
Yea happy to do that! I’ll probably run it when I’m back on my work computer tomorrow though
y

Yoni Augarten

10/09/2022, 6:16 AM
Basically, my thought is that you can clear up any multipart uploads from before a certain point.
DELETE from gateway_multiparts WHERE creation_date < '2022-09-01';
But let's ask @Itai Admi if this solution makes sense.
i

Itai Admi

10/09/2022, 7:58 AM
It might help, it depends if the null value is before or after the filter date. I much rather we “really” fix it by changing the migration to support these null values. Taking a look and updating here.
Created a PR to fix it - it will skip null content-type during migration as it was a bug in the way we added content-type to multiparts uploads. Users can’t use these stale (originated before v0.53.0) multiparts today anyhow - kv just exposed the bug. Once the PR is approved I’ll release a lakeFS version v0.80.2 that fixes the migration.
Hi @Thomas Vander Wal - lakeFS 0.80.2 was released with a fix to the migration. Can you try running the migration again using this version?
:heart_lakefs: 3
t

Thomas Vander Wal

10/09/2022, 7:57 PM
Thank you @Itai Admi and @Yoni Augarten! Using
0.80.2
in the migrate job did the trick. I appreciate the help 🙂
🙌 1
🙌🏼 1
y

Yoni Augarten

10/09/2022, 8:00 PM
You're welcome!🙂