Is there a way to stop lakefs choking on concurrent branch merges?

Yes! Try setting

graveler.branch_ownership.enabled

to

true

(of course as with any config option you could use

LAKEFS_GRAVELER_...

). This is an experimental feature - but it is safe in the sense that it cannot give incorrect results. If you use it, please do let us know of your experiences - maybe one day it will stop being experimental.

@Ariel Shaqed (Scolnicov) so LAKEFS_GRAVELER_BRANCH_OWNERSHIP_ENABLED="true" for the env?

Yup!

Ok and this for the config, right:

graveler:
    branch_ownership:
      enabled: true

Sure!

What does this experimental feature actually do? 😛

It kinda locks branches for mutating branch operations (commits and merges of all types). Now of course locks are anathema in any distributed system. So instead it uses something we call "weak ownership": this mostly ensures only a single thread on the entire lakeFS cluster will own the branch, but sometimes another thread can get in. Because mutating branch operations are safe in lakeFS even when they race, such an ownership failure is safe: it will give slower results, but not incorrect results. (In practice I have never seen such a failure. But of course that doesn't matter in a distributed system like lakeFS. What does matter is that there is no race in these operations, regardless of whether or not you enable the feature.)

Alright, so we don't need to retry anymore but just await for a lock release

Alright, so we don't need to retry anymore but just await for a lock release

Yeah pretty much. The speedup happens because the lakeFS servers no longer waste iops on creating commits which will fail. The fact that there are many fewer retries is because there are many fewer conflicts - it's not the goal.

I think I might add this to our deltalake lakfes integration docs

I understand the advantages, and as you can tell I really like the feature. But equally I cannot stress too highly: this is an experimental feature. I would ask that delta-rs docs describe it as such. It does not have the maturity which we associate with the rest of core lakeFS. So it might go away with no warning. Alternatives for operators are to reduce parallelism, increase KV performance, or add more lakeFS pods. Each has its own advantages, and almost all of these can and should be combined. What can I say - at some point getting high throughput becomes hard.

If there is merge conflict that indicates the delta commit was already merged, so we increment the version with and then retry merge

Sorry, 429 I guess. But also 409: you will fail if you retry the operation on lakeFS. Indeed you "_increment the version_ ... and then retry merge". As far as lakeFS is concerned, that's not a retry - it's a different merge. 😇