https://lakefs.io/ logo
#help
Title
# help
p

Paul

05/09/2023, 11:47 AM
Hello, I have a question about the storage capacity of lakefs, what is the maximum size of a repo?
i

Isan Rivkin

05/09/2023, 12:22 PM
Hey @Paul, there is no predetermined limit. We’ve had repo’s with 100Ms of objects. We have repos with 10Ks of versions. We’re managing metadata, so lakeFS doesn’t care about total size of all objects in a repo.
p

Paul

05/09/2023, 12:51 PM
I understand this but when I looked my lakeFS repository on the UI, there isn't just metadata but all my images are here and viewable. So how it's possible if lakeFS doesn't copy data while importing them ?
i

Iddo Avneri

05/09/2023, 1:09 PM
Hi @Paul, when you import the images, lakeFS directs you to their original location. You can learn more here: https://lakefs.io/blog/best-practices-to-adopt-lakefs/
p

Paul

05/10/2023, 7:50 AM
Thanks for the answer. So just for information, what is the difference between metadata store in lakeFS folder on the S3 bucket and metadata store in postgreSQL database (on kv table) ?
i

Isan Rivkin

05/10/2023, 8:19 AM
Hi @Paul, in the context of lakeFS you can read more about supported storage and Architecture here, we store some metadata related to a repository on the storage (i.e s3) itself just like
.git
folder. We manage metadata references with Graveler and support out-of-the-box DynamoDB and PostgreSQL. You can read about the tradeoffs here between them.
p

Paul

05/10/2023, 9:40 AM
I've read this doc about metadata objects but if I well understand : metadata store in PostgreSQL are commit, repo, tags... So my question just is what is in lakeFS metadata which are encoded with sorted strings table ?
I mean file created on S3 bucket with 64 hexadecimal characters for name
i

Isan Rivkin

05/10/2023, 9:56 AM
Part of what I mentioned above is Graveler this should answer your SST question🙂
p

Paul

05/10/2023, 12:52 PM
Thanks @Isan Rivkin for your response and the link. So, lakefs storage is separated with range and metarange objects. My first question is : What is min_key value in metarange object? Other question, if I do an import I can see two files in lakefs folder on my bucket but if I do an import and then a commit I supose that I will see three files? One metarange, one range for import command and another for merge command, it's true?
i

Isan Rivkin

05/10/2023, 1:02 PM
The Meta-ranges/ Ranges are not files that you see in the repo itself, in the lakeFS folder yes, they are all internal implementation details, If you want to learn more in-depth about Meta-Ranges / Ranges and everything related to object-store address translation you can watch this

youtube talk

.
20 Views