Hey everyone, just wanted to share some concerns about LakeFS (Version 1.29.0) 1. Is LakeFS removing from S3 the folder created when a repository is deleted? a. If not, why? I mean, LakeFS is a data versioning tool, if we are keeping data that was potentially removed by the user why are we keeping that in S3 2. Removing a repository make that name not usable anymore (I suppose this is coming by the concern explained above) 3. When I upload the same object to LakeFS (without any change), store the object again, taking up storage space (for small object this is not a big deal but since people normally saves here data and the common usage is to upload the folder directly, not the edited files only) 4. Creating Tags consume storage space?

1. Regardless of a deleted repo, a lakeFS server never deletes data from the object store. A lakeFS Garbage Collection (a Spark jar) does delete data. We had several discussion around this, tl;dr deleting the data from the object store during repo deletion is both irreversible and also might not scale in some cases. 2. Using the same storage namespace of a different repo is not valid. Using the same repo name should be valid after a short (5-10s) time of metadata cleanup in the server side. 3. Deduplication of objects isn’t available with lakeFS. Garbage collection is in charge of removing files that were overidden or not being pointed by any branch according to the branch retention policies. 4. Creating tags is a metadata only operation which does not create any objects in the object store.

Thanks @Itai Admi. So if I configure the Garbage policies, extra data and removed repos will be removed from the bucket?

Please read about Garbage Collection in our docs. Basically, it’s a Spark job you need to run for cleanup of data.

thanks!