We have deployed dataset to a local lakefs (docker). Question: later on, we will have a lakefs deployed in azure. How do you migrate the data and history to the new lakefs server?

Hi @HT, welcome to the lake! :lakefs: Are you planning on using the same storage with a new lakeFS deployment or did you intend to migrate both the storage and lakeFS?

Both. Current storage is local disk. Later on will be azure blob storage

lakeFS currently has no feature to support migration from different block stores. If you have a relatively small repository it might be possible to achieve this using the export / import functionalities and some scripting. Note that you will have to build your repos and branches manually and then export - import each commit separately

Thanks. Good to know this in advance. I will a deeper look. Thank you

Hi @HT, want to suggest an alternative way to move your environment. It assume that you didn't import any data while working with local - only uploaded data. We can use lakectl's ref-dump command and restore the repository information (commit all changes before dump). Copy the local data to Azure and override the repository storage namespace to the new location. The commits, tags and branch information should be kept.

No we don't have imported data.

Steps - per repository 1. Commit changes 2. ref-dump 3. copy the lakefs data to azure blob store - rclone 4. create bare repository point to the azure blob store 5. refs-restore

Will give it a try with 2 local lakefs. It's easier to setup ;)

@HT did you create a bare repository or a regular repository?

@HT About step 3 - copy the data to azure. Its my bad, it is probably should be swapped with step 4 - create bare repository. When you create a repository you select the location you like to keep lakeFS metadata and data.

@Barak Amar Success ! I manage to move one repo from one local server to another. There are 2 questions : 1. How do you delete a bare repo ?? 2.

lakectl refs-dump

output something like :

Repository: <lakefs://tomato>

{
  "branches_meta_range_id": "41dcb9faf1495bf6b3915ce5c18c44eb74cd88ba076459ac2f3348992197b537",
  "commits_meta_range_id": "175ab55e94e8ad486c30c3967653202de305d8b0dd01bffd75e4aae57578a2f4",
  "tags_meta_range_id": "e77ccbca0369cb68eb14455302ee10f4baaf7aff716ec1b591aeb9b86b198d19"
}

While the json is only :

{
  "branches_meta_range_id": "41dcb9faf1495bf6b3915ce5c18c44eb74cd88ba076459ac2f3348992197b537",
  "commits_meta_range_id": "175ab55e94e8ad486c30c3967653202de305d8b0dd01bffd75e4aae57578a2f4",
  "tags_meta_range_id": "e77ccbca0369cb68eb14455302ee10f4baaf7aff716ec1b591aeb9b86b198d19"
}

This looks like a bug ...

1. same as any other repo. btw it is no longer bare after we restore the refs. 2. yes. thank you! if you can open an issue and we will address this one.

Thanks ! Was going to log an issue but then the issue been fixed when i woke up ! Thanks !

tz differences 🙂

re: deleting bare repo. I could not delete them from the WebUI. I forgot about the CLI. I managed now to delete them from CLI.

Thanks @HT. Does that mean you are good to go?

Yes. I managed to delete and clean up the bare repo

Excellent. 🙏