I have another question related to using delta in combination with lakefs after reading the limitations section:

The Delta log is an auto-generated sequence of text files used to keep track of transactions on a Delta table sequentially. Writing to one Delta table from multiple lakeFS branches is possible, but note that it would result in conflicts if later attempting to merge one branch into the other. For this reason, production workflows should ideally write to a single lakeFS branch that could then be safely merged into main.

Is this also true when the table is concurrently updated by different processes, but they work on different partitions? Ideally I would like to create one branch per writer and merge into main when the write finishes the data processing.

Hey @Alessandro Mangone! That's a good question - the Delta log would conflict even if different, unrelated partitions are modified. The use case you're describing makes total sense and is currently planned on our roadmap - we are looking for design partners to get their feedback on proposed solutions and to try out early releases - would love your help with that!