Hey Everyone, What is the suggested way for migration of data from one lakeFS installation to other? Looking forward to the answers 🙂

Hey @mishraprafful - That's a good question!

Hi @Oz Katz yes, we are looking at transferring all the repos to a different installation.

but you do plan on setting up a new PostgreSQL server?

yes, most probably that would change as well. We are planning to migrate PostGres and S3 (or keep them the same), what other points of failure do you think might affect this migration ?

I guess the safest option would then be to setup the second environment and migrate the repos one by one, for each repo: 1. on the old installation, run

lakectl refs-dump

(see reference): this writes all metadata related to the repository (commits, branches, tags) to the repository’s storage namespace 2. on the new installation - create a bare repository with the same name 3. copy the old repository’s storage namespace to the new one's location using e.g. rclone 4. run

lakectl refs-restore

(reference) to load back all commits, tags and branches 5. once it's all done, switch clients to the new repo. one note however, is that currently, refs-dump doesn't handle uncommitted records, so make sure to commit any staged writes prior to migration

Thanks a lot for the suggestions @Oz Katz. I think we will go with the first one most probably.

Great! feel free to jump back here with any question..

Sure, thanks 🙂