Files are then eagerly fetched and then locally available. After this there is no leakage of S3/GCS into the training code. I was hoping to do something where I abstracted away that into say a LakeFS Dataset where a user could simply pass in the name of a dataset aka branch. So something like ...
However, we are extremely sensitive to latency and have many, many researchers which combinatorially suggests the need to run many instances of the S3 Gateway. 1. I want to understand the design choice behind not simply allowing the user the ability resolve those paths, both for my own understanding and to... 2. to justify the cost and complexity. We are quite savy with both K8's and Cloud in general, so it isn't that we cannot run the S3 Gateways, but we need to understand the reasoning in order to make a good comparison against other options