Hello, I have a question about the storage capacity of lakefs, what is the maximum size of a repo?

Hey @Paul, there is no predetermined limit. We’ve had repo’s with 100Ms of objects. We have repos with 10Ks of versions. We’re managing metadata, so lakeFS doesn’t care about total size of all objects in a repo.

I understand this but when I looked my lakeFS repository on the UI, there isn't just metadata but all my images are here and viewable. So how it's possible if lakeFS doesn't copy data while importing them ?

Hi @Paul, when you import the images, lakeFS directs you to their original location. You can learn more here: https://lakefs.io/blog/best-practices-to-adopt-lakefs/

Thanks for the answer. So just for information, what is the difference between metadata store in lakeFS folder on the S3 bucket and metadata store in postgreSQL database (on kv table) ?

Hi @Paul, in the context of lakeFS you can read more about supported storage and Architecture here, we store some metadata related to a repository on the storage (i.e s3) itself just like

.git

folder. We manage metadata references with Graveler and support out-of-the-box DynamoDB and PostgreSQL. You can read about the tradeoffs here between them.

I've read this doc about metadata objects but if I well understand : metadata store in PostgreSQL are commit, repo, tags... So my question just is what is in lakeFS metadata which are encoded with sorted strings table ?

Part of what I mentioned above is Graveler this should answer your SST question🙂

Thanks @Isan Rivkin for your response and the link. So, lakefs storage is separated with range and metarange objects. My first question is : What is min_key value in metarange object? Other question, if I do an import I can see two files in lakefs folder on my bucket but if I do an import and then a commit I supose that I will see three files? One metarange, one range for import command and another for merge command, it's true?

The Meta-ranges/ Ranges are not files that you see in the repo itself, in the lakeFS folder yes, they are all internal implementation details, If you want to learn more in-depth about Meta-Ranges / Ranges and everything related to object-store address translation you can watch this .