Hey all I ve got some questions about migrating repos betwee lakeFS #help

Hey all, I’ve got some questions about migrating r...

Jacob

10/17/2023, 2:36 PM

Hey all, I’ve got some questions about migrating repos between instances https://lakefs.io/blog/tutorial-migrate-or-clone-lakefs-repository/ 🧵

Jacob

10/17/2023, 2:37 PM

1. In step 3, is it possible to just sync a single repo? The aws s3 command is syncing the whole bucket

Jacob

10/17/2023, 2:38 PM

2. In step 2, can refs-dump be called multiple times? For example, I dump the refs but have to delay the migration and retry later?

Jacob

10/17/2023, 2:39 PM

3. Can this process be called multiple times if new data lands on the source repo after migrating? Or would I need to delete the target repo entirely and start over?

Jonathan Rosenberg

10/17/2023, 2:56 PM

Hi @Jacob 1. It’s possible to sync a single repo by providing the storage namespace in which you repo resides. The

aws s3 sync

command is not limited to buckets, but can accept paths in them. So if your repo’s storage namespace is

<s3://my-bucket/my-repo>

you can use

aws s3 sync <s3://my-bucket/my-repo>

<s3://my-other-bucket/my-other-repo>

. 2. Yes. 3. Since you need a bare (clean) repository at the target repo environment, you would need to clear the target environment. Can you describe the use case for which you would need to migrate your repos?

Jacob

10/17/2023, 6:36 PM

Thanks for confirming! We need to migrate some repos to a lakefs deployment in a different cluster

Jacob

10/17/2023, 6:41 PM

@Jonathan Rosenberg will refs-dump cause any changes to the source repo if we run it but then decide not to migrate?

Jonathan Rosenberg

10/17/2023, 6:41 PM

No It only saves a copy of the DB

🙏 1

9 Views

Open in Slack

Previous Next