Hello, LakeFS community. Is there a way to mirror a LakeFS instance in another region? Specifically, I have a LakeFS instance connected to a DynamoDB global table and an S3 bucket. And I have another LakeFS instance connected to the same DynamoDB global table. The second LakeFS instance uses the same S3 instance. But I want it to use an S3 bucket that is a bidirectional replica of the other S3 bucket. Is this possible? The obvious answer would be to specify a multi-region access point instead of an S3 bucket. But LakeFS doesn't seem to support using a multi-region access point. Am I missing something here? For example, would this architecture be unstable due to replication delays?

Hi @CC, There is no current support for replicated lakeFS. Generally lakeFS tries to provide an interface similar to other object stores -- which are also non-replicated. And it provides very strict semantics for concurrency, because that is required for versioning. Trying to replicate objects between buckets in different regions is intriguing, but off-hand I am not sure how this would work. For instance, suppose that i have lakeFS instances that try to mirror one another, one in us-west-2 one in eu-central-1: what would you expect to happen if I concurrently commit on both? (I am not sure how multi-region access points would work to solve the issues with concurrent operation semantics on different lakeFS instances; I do see how it might route operations to the right lakeFS instance... but that is probably the smaller issue.)

I agree that concurrent commitment would be problematic for any dual-master situation. AWS offers last-request-wins for replicating between 2 buckets, and assigning numeric priorities for replicating between more than 2 buckets. But AWS is solving a different problem, and their solution may very well fail for LakeFS replication. However, a master-slave arrangement (one-way replication) would be extremely useful, and seems more technically feasible for LakeFS. For some reason, I thought LakeFS Cloud would be capable of replicating (one-way or two-way) across regions. If practical, that could be quite useful. But it sounds like that was wishful thinking on my part? (It's not for me to say what should and should not be technically feasible. Just clarifying the current capabilities.)

Thanks! IIUC you're asking for a multiregion readonly replica. I'd be really grateful if you could open an issue to do that. Could you also add your use case? Mostly in terms of volume of writes and reads - whatever we do has to be cheaper than cross-regional access, or we gain little. Have you considered exporting the branch to s3? Because atomicity will anyway be weaker across replicas...

What is the process for opening an issue? The problem with exporting a branch is that it nullifies many of the reasons for using LakeFS in the first place: 1) The deduplication from branching is lost. 2) The redundant files then have to be transferred across regions. 3) The commit ID loses its significance in the target S3. All of those could be addressed by copying to another LakeFS instance, using rclone. But it feels like the complexity is going up pretty fast. Is there a way to forward (really just copy) all requests to one LakeFS instance, so they also go to another LakeFS instance?

Issues are just https://github.com/treeverse/lakeFS/issues/new But I'm afraid I don't understand the architecture you'd like to use: if data and metadata are stored at a remote region, having just a local lakeFS might not make much of a difference (for use cases I have in mind, at least). Would you be willing to hop on a VC to discuss? Please message me privately if you can and we'll set up something. Thanks!