Regarding caching data on the lakeFS server, lakeFS is designed to work with very big objects. In particular, caching them in memory or on disk is assumed not to be feasible.
11/21/2022, 5:26 PM
I can see the rationale behind that, and of course I know little about the average user of lakefs. But especially for DL use cases, where you might only access one of many datasets&versions at the same time, but these again and again, I could imagine that especially for hybrid setups (i.e. data saved in different clouds/cloud regions or on-prem) it might be beneficial to have such a capability for low latency, high speed, colocated access to the objects without the need for any of the durability guarantees (which would be provided by the object storage) and the low-cost archival of old versions.
11/21/2022, 5:40 PM
Thank you for elaborating. I can see your point. I will discuss this with the team as well.