https://lakefs.io/ logo
Join Slack
Powered by
I’ve been upgrading our lakeFS instances to the ne...
# help
t
I’ve been upgrading our lakeFS instances to the newest version but having issues on our main one. Error in thread but the 0.80.0 migration is failing due to a null field issue, which makes me think something may have gotten out of sync inside the database.
Copy code
time="2022-10-09T05:09:17Z" level=info msg="Migrated 100000 entries" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" created_at="2022-10-09 05:09:03.593783788 +0000 UTC" db_schema_version=1 lakefs_version=0.80.0 package_name=graveler
time="2022-10-09T05:09:27Z" level=info msg="Migrated 200000 entries" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" created_at="2022-10-09 05:09:03.593783788 +0000 UTC" db_schema_version=1 lakefs_version=0.80.0 package_name=graveler
time="2022-10-09T05:09:38Z" level=info msg="Migrated 300000 entries" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" created_at="2022-10-09 05:09:03.593783788 +0000 UTC" db_schema_version=1 lakefs_version=0.80.0 package_name=graveler
time="2022-10-09T05:09:49Z" level=info msg="Migrated 400000 entries" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" created_at="2022-10-09 05:09:03.593783788 +0000 UTC" db_schema_version=1 lakefs_version=0.80.0 package_name=graveler
time="2022-10-09T05:10:00Z" level=info msg="Successfully migrated package to KV" func=pkg/kv/postgres.Migrate.func1 file="build/pkg/kv/postgres/migrate.go:129" pkg_id=graveler
time="2022-10-09T05:10:00Z" level=info msg="graveler Package Migration took 56.506001978s" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" TempDir=/tmp/kv_migrate_4110592530
time="2022-10-09T05:10:00Z" level=info msg="KV Migration took 56.506086254s" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:246" TempDir=/tmp/kv_migrate_4110592530
Failed to setup DB: 1 error occurred:
   * failed migration on package multiparts: scany: scan row into struct fields: can't scan into dest[5]: cannot scan null into *string
y
Hey @Thomas Vander Wal, can you please specify from and to which version you are upgrading?
t
Hey Yoni! I’m currently on 0.66.0 and going to to 0.80.0 for the migration, before upgrading to 0.83.2 This did work for another instance that was on 0.63.0
y
Thank you, I'm checking it out and will get back to you shortly
heart lakefs 1
t
It does look like the 0.69.0 migration did succeed so I can confirm my issue is with the latest
y
@Thomas Vander Wal, do you mean a migration from or to 0.69.0?
t
To 0.69.0. The only way I was able to get the service back up and running was to switch to that version.
y
I see. I'm trying to reproduce the issue on my end. Will update here
It seems like there is an issue with the migration process, it is related to a database table that keeps track of multipart uploads into lakeFS. @Thomas Vander Wal, to unblock you, I can suggest a query to run directly on the database before upgrading. Is that something you can do?
t
Yea happy to do that! I’ll probably run it when I’m back on my work computer tomorrow though
y
Basically, my thought is that you can clear up any multipart uploads from before a certain point.
Copy code
DELETE from gateway_multiparts WHERE creation_date < '2022-09-01';
But let's ask @Itai Admi if this solution makes sense.
i
It might help, it depends if the null value is before or after the filter date. I much rather we “really” fix it by changing the migration to support these null values. Taking a look and updating here.
Created a PR to fix it - it will skip null content-type during migration as it was a bug in the way we added content-type to multiparts uploads. Users can’t use these stale (originated before v0.53.0) multiparts today anyhow - kv just exposed the bug. Once the PR is approved I’ll release a lakeFS version v0.80.2 that fixes the migration.
Hi @Thomas Vander Wal - lakeFS 0.80.2 was released with a fix to the migration. Can you try running the migration again using this version?
heart lakefs 3
t
Thank you @Itai Admi and @Yoni Augarten! Using 
0.80.2
in the migrate job did the trick. I appreciate the help 🙂
🙌 1
🙌🏼 1
y
You're welcome!🙂
11 Views
Open in Slack
PreviousNext
Powered by