Hi, does Lakefs have a limit on the number of repositories? I ask this question because I noticed that the pgsql implementation is configured with 100 partitioned tables, and the data related to each repository is stored in the same partitioned table. Therefore, I am unsure whether having a large number of repositories would cause any additional issues or side effects. Additionally, what are the benefits of storing all data under the same table structure rather than using different tables? Would using different tables potentially reduce serialization overhead?

Hi Yuhao, Internally lakeFS needs a KV store with fairly minimal concurrency guarantees. The Postgres backend is perfectly good, but for high capacity or high throughout I would probably recommend using a cheaper option - say DynamoDB or CosmosDB on a cloud. If you use lakeFS Cloud, we manage it for you. In any case, the load on the database will probably be due to managing the staging area - uncommitted data. So it will be most affected by the rate of operations on the most active branch. I don't know you use case of course. But I would be surprised if the number of repositories significantly affected performance.

and for the sake of completeness, adding to Ariel’s detailed answer - there’s currently no lakeFS limit on the number of repositories.

@Ariel Shaqed (Scolnicov) So can I assume that this table storage structure does indeed affect performance (For example, serialization), but it is compatible with other key-value storage systems.

Yes, it will probably affect performance.