Title
#help
t

Thomas Vander Wal

10/09/2022, 5:20 AM
I’ve been upgrading our lakeFS instances to the newest version but having issues on our main one. Error in thread but the 0.80.0 migration is failing due to a null field issue, which makes me think something may have gotten out of sync inside the database.
5:21 AM
time="2022-10-09T05:09:17Z" level=info msg="Migrated 100000 entries" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" created_at="2022-10-09 05:09:03.593783788 +0000 UTC" db_schema_version=1 lakefs_version=0.80.0 package_name=graveler
time="2022-10-09T05:09:27Z" level=info msg="Migrated 200000 entries" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" created_at="2022-10-09 05:09:03.593783788 +0000 UTC" db_schema_version=1 lakefs_version=0.80.0 package_name=graveler
time="2022-10-09T05:09:38Z" level=info msg="Migrated 300000 entries" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" created_at="2022-10-09 05:09:03.593783788 +0000 UTC" db_schema_version=1 lakefs_version=0.80.0 package_name=graveler
time="2022-10-09T05:09:49Z" level=info msg="Migrated 400000 entries" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" created_at="2022-10-09 05:09:03.593783788 +0000 UTC" db_schema_version=1 lakefs_version=0.80.0 package_name=graveler
time="2022-10-09T05:10:00Z" level=info msg="Successfully migrated package to KV" func=pkg/kv/postgres.Migrate.func1 file="build/pkg/kv/postgres/migrate.go:129" pkg_id=graveler
time="2022-10-09T05:10:00Z" level=info msg="graveler Package Migration took 56.506001978s" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" TempDir=/tmp/kv_migrate_4110592530
time="2022-10-09T05:10:00Z" level=info msg="KV Migration took 56.506086254s" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" TempDir=/tmp/kv_migrate_4110592530
Failed to setup DB: 1 error occurred:
   * failed migration on package multiparts: scany: scan row into struct fields: can't scan into dest[5]: cannot scan null into *string
Yoni Augarten

Yoni Augarten

10/09/2022, 5:27 AM
Hey @Thomas Vander Wal, can you please specify from and to which version you are upgrading?
t

Thomas Vander Wal

10/09/2022, 5:28 AM
Hey Yoni! I’m currently on 0.66.0 and going to to 0.80.0 for the migration, before upgrading to 0.83.2 This did work for another instance that was on 0.63.0
Yoni Augarten

Yoni Augarten

10/09/2022, 5:29 AM
Thank you, I'm checking it out and will get back to you shortly
t

Thomas Vander Wal

10/09/2022, 5:45 AM
It does look like the 0.69.0 migration did succeed so I can confirm my issue is with the latest
Yoni Augarten

Yoni Augarten

10/09/2022, 5:53 AM
@Thomas Vander Wal, do you mean a migration from or to 0.69.0?
t

Thomas Vander Wal

10/09/2022, 5:53 AM
To 0.69.0. The only way I was able to get the service back up and running was to switch to that version.
Yoni Augarten

Yoni Augarten

10/09/2022, 5:54 AM
I see. I'm trying to reproduce the issue on my end. Will update here
6:12 AM
It seems like there is an issue with the migration process, it is related to a database table that keeps track of multipart uploads into lakeFS. @Thomas Vander Wal, to unblock you, I can suggest a query to run directly on the database before upgrading. Is that something you can do?
t

Thomas Vander Wal

10/09/2022, 6:13 AM
Yea happy to do that! I’ll probably run it when I’m back on my work computer tomorrow though
Yoni Augarten

Yoni Augarten

10/09/2022, 6:16 AM
Basically, my thought is that you can clear up any multipart uploads from before a certain point.
DELETE from gateway_multiparts WHERE creation_date < '2022-09-01';
But let's ask @Itai Admi if this solution makes sense.
Itai Admi

Itai Admi

10/09/2022, 7:58 AM
It might help, it depends if the null value is before or after the filter date. I much rather we “really” fix it by changing the migration to support these null values. Taking a look and updating here.
9:27 AM
Created a PR to fix it - it will skip null content-type during migration as it was a bug in the way we added content-type to multiparts uploads. Users can’t use these stale (originated before v0.53.0) multiparts today anyhow - kv just exposed the bug. Once the PR is approved I’ll release a lakeFS version v0.80.2 that fixes the migration.
10:55 AM
Hi @Thomas Vander Wal - lakeFS 0.80.2 was released with a fix to the migration. Can you try running the migration again using this version?
t

Thomas Vander Wal

10/09/2022, 7:57 PM
Thank you @Itai Admi and @Yoni Augarten! Using
0.80.2
in the migrate job did the trick. I appreciate the help 🙂
Yoni Augarten

Yoni Augarten

10/09/2022, 8:00 PM
You're welcome!🙂