Alessandro Mangone11/29/2022, 11:27 AM
Is this also true when the table is concurrently updated by different processes, but they work on different partitions? Ideally I would like to create one branch per writer and merge into main when the write finishes the data processing.
The Delta log is an auto-generated sequence of text files used to keep track of transactions on a Delta table sequentially. Writing to one Delta table from multiple lakeFS branches is possible, but note that it would result in conflicts if later attempting to merge one branch into the other. For this reason, production workflows should ideally write to a single lakeFS branch that could then be safely merged into main.
Oz Katz11/29/2022, 11:47 AM