Hello team, I have a LakeFs server running in a A...
# help
v
Hello team, I have a LakeFs server running in a AWS ECS cluster behind an ALB. If two lambada function tries to write the same file in same branch in same repository, using lakefs python clinet library, is lakeFs able to store writes from both lambdas. Will I loose the data written from one lambda as the other lambda overrides them.
i
Regardless of your AWS setup, last write to the same file wins. Probably a silly question, but why not write to different files if both inputs are valuable? Notice that if you commit between 2 writes, then you’ll see the first write on the commit and the last write in the uncommitted section of the branch.
v
Thank you. One file simplifies our reading strategy. That's why I was wondering whether there is any locking mechanism in LakeFs. But I agree on your suggestion.
a
IIUC, two lambdas write to the same path concurrently. What would your desired result be in such a case? An example: as an alternative to @Itai Admi’s suggestion, if you are using the high-level Python SDK then you might use Branch.transact in each Lambda, and retry on conflict.