Hi
@Barak Amar and VE-team,
Really excited to see
lakeFS on KV is now a proposal! Did you incorporate my "lock freedom" suggestions on the PR into the proposal? These are intended to ensure that at all times at least one commit makes progress.
These will be
really important in order to support streaming workflows.
For instance, for almost any use of Kafka with lakeFS, we will have periodic commits which may contain many objects. As soon as there is a hiccup we will have concurrent commits -- all of these after the first large commit will also be large! Without lock freedom, concurrent commits get in one another's way, and progress might not occur. As more commits pile up, the probability of any one of them succeeding drops rapidly, and the system will have unbounded delays and might effectively livelock.