Thread
#lakefs-for-beginners
    Adi Polak

    Adi Polak

    2 weeks ago
    Heya, I am looking to better understand lakeFS data model with DynamoDB. when we think about consistency and atomic operations, where is the locking mechanism taking place? and how to ensure atomic operations are happening when there is a need to scale the lakeFS server?
    Itai Admi

    Itai Admi

    2 weeks ago
    lakeFS data model has not changed with the transition to the
    kvstore
    . The
    kvstore
    simple interface allows many different databases to implement it, currently we have DynamoDB and Postgres. We use the
    SetIf
    functionality to ensure that operation that started on a given entity (e.g. branch) will only succeed if the entity didn’t change since it was read (e.g. multiple operations trying to change the branch HEAD). Other than that we must configure the DB in a way that will ensure read-after-write consistency, luckily DynamoDB has that option.
    Adi Polak

    Adi Polak

    2 weeks ago
    thanks, @Itai Admi. so as far as I understand, the lock is managed by the key-value store, and since DynamoDB supports read-after-write strong consistency, it is by design makes lakeFS branch operations atomic. would it be possible to leverage s3 as kV store as well? in 2020 s3 introduce a strong read after writing and by design, it is a key-value store. or could it be that because of s3 naming collision that might introduce problems in the future with key names?
    Itai Admi

    Itai Admi

    2 weeks ago
    The
    SetIf
    functionality is a fancy name for
    compare-and-swap
    . There’s no easy way to achieve this with S3, so the answer is unfortunately not at this moment.
    Adi Polak

    Adi Polak

    2 weeks ago
    so basically passing the predicate - compare - and if predicate, then, swap? so the dynamodb API itself is PutItemWithContext?
    Itai Admi

    Itai Admi

    2 weeks ago
    The DynamoDB API call is PutItem (
    PutItemWithContext
    is the golang sdk api). We use ConditionExpression to compare the blobs. Code reference.
    Adi Polak

    Adi Polak

    2 weeks ago
    thanks!!