https://lakefs.io/ logo
Join Slack
Powered by
Hey there everyone, I just need some help with my...
# help
c
Hey there everyone, I just need some help with my recently installation of LakeFS. We want to automatically create backups of the repositories metadata, so if any time the docker is removed, we can restore that. I have seen from lakefs_sdk that exists dump_submit, but I don’t know if it creates a json and where. Could you share your ideas? 🙂
i
Hi Carlos, You can take a look at this project which has multiple examples of how to migrate and clone repositories using lakeFS python SDK. It contains multiple examples of different storage types. You can also look at this article which has an example of how to achieve that by using lakectl client. Notice that ref-dump only backup your metadata and not the data Please let me know if that's helpful and if you managed to back up your repositories.
c
Hey! thanks for your quick response! I will try that and let you know! thanks mate.
jumping lakefs 1
@Idan Novogroder I have seen some messages here complaining about ref-dump not creating the refs_manifests in e.g. s3 (aws). I looked in the bucket in _lakefs and there are only the refs (branch, tags and other file cant remember the name) but no refs_manifest
Copy code
{
  "branches_meta_range_id": "700ec8fdecf86c45b25f3767a5422496f9ccccdf918cbc157a6953fc8",
  "commits_meta_range_id": "4ed6d35e5bd4206f2285a5f2274c701df554dbf774c7e6fabffa936ae",
  "tags_meta_range_id": "7cc6abb06825ee467df1b3a92a8109c5bf14d1044d217ed0595fe00c8a"
}
is that json the manifest? by executing the command it just creates the files in S3 and print the manifest (not storing)
i
Yes, this is the manifest 🙂
a
Hi @Carlos Luque, I would guess that you are running lakeFS with its database stored in a container on Docker. If so, and in addition to what Idan said, I would strongly urge you not to use Docker like this in a production system. A database should be a stable system. If you lose it, you are guaranteed to lose all your work in progress, including everything that was uploaded to lakeFS but not committed. There are many options for getting a reasonable database, including hosted Postgres (such as Aurora or RDS on AWS), a properly maintained local database, hosted KV stores such as CosmosDB or DynamoDB, or of course just migrate over to lakeFS Cloud.
👍🏽 1
👍 1
c
Yes, we are running the lakeFS on Docker, I’ll try what you suggest! Thanks!
Open in Slack
PreviousNext
Powered by