Hi all, I'm experiencing an unusual behavior with garbage collection, and I need some assistance. Below is the current configuration:

{
  "branches": [
    {
      "branch_id": "main",
      "retention_days": 7
    }
  ],
  "default_retention_days": 1
}

However, I've noticed that some branches are much older than 1 day, even though the retention period should be respected. Can you help me understand what might be causing this behavior? Is there a known issue in this area? Thank you for your help! server version: 1.5

Hi Dima, The GC should delete objects found in commits that are no longer referenced by the main branch after 7 days (based on the commit time), the default retention for other branches is 1 day. However, if they point to the same commit as one used by the main branch, the data will be kept. Following a generated Python script that can help you graph the commits from main and other branches. This will help visualize the commits with their timestamps to help identify if we have an issue with processing the correct commits. The script requires lakefs_sdk and pyyaml.

Hi Barak, Thanks for the quick response. If I understand correctly, there was a merge between two branches, and as a result, the branches will remain together from now on. Is there a solution to handle this issue, as we are currently storing too much data? Any suggestions would be greatly appreciated.

The merge between the two is an option, if we look at the graph we can try and explain why there are things you see that GC doesn't address. Yes, merge generates a commits - you can merge into main and from main. For the question above, we need to understand which commits the main branch reach and which commits the 'other' branch reaches to understand if there is an issue. If you find a commit on 'other' branch, older than 1 day that your main branch does not reach in 7 days (commits log) and a file is no longer reached by that commit, we can try and dig deeper. If the data is reachable so based on the rules we should keep it.

Hi @Dima Paikin, From your message I'm under the impression that you think the GC process delete branches. If that's not the case please ignore my message. GC delete objects from the object store based on branch retention rules according to the logic Barak described and it does not delete branches

Hi @Niro,@Barak Amar, and everyone, Thanks for the helpful insights so far. I think I was a bit confused between the data associated with the branch and the branch itself (kind of the metadata aspect). My questions are mainly from a cost perspective, and I have three clarifications I’d like to request: 1. How can I be sure that after merging a branch into the master, there is now only a single instance of the data, rather than multiple references to the same data across different branches? 2. In cases where a branch wasn’t successfully merged into master (e.g., due to a job failure), how can I be sure that the data will still be properly cleaned up after the retention period expires (e.g., 1 day)? 3. When I view the branches menu in the UI, is there any way to distinguish between a branch that was successfully merged into master and a "zombie" branch (i.e., one that wasn’t merged and should eventually be cleaned up)? And the bonus question: From my perspective, there’s no real value in keeping merged branches visible. Is it possible to configure the system so that merged branches (as metadata) are erased automatically, similar to how GitHub removes them after merging to master? Looking forward to your thoughts!

Hi Dima, I'll try to address each one: How can I be sure that after merging a branch into the master, there is now only a single instance of the data, rather than multiple references to the same data across different branches? Merge to the branch produce a commit that references the merged data from the two branches. We don't copy data at this point the merged commit points to data produced by both branches. (or one/none - depends on the change) Any branch that reference the commit (source or merge) points to the same data. In cases where a branch wasn’t successfully merged into master (e.g., due to a job failure), how can I be sure that the data will still be properly cleaned up after the retention period expires (e.g., 1 day)? Merge doesn't produce data, so the commit still exists and referenced by the original branch. When I view the branches menu in the UI, is there any way to distinguish between a branch that was successfully merged into master and a "zombie" branch (i.e., one that wasn’t merged and should eventually be cleaned up)? The branch is not a zombie, as it can be listed in the system and points to a commit. In the commits tab when selecting 'main' branch you see the merge commit "Merge 'X' into 'main'". From my perspective, there’s no real value in keeping merged branches visible. Is it possible to configure the system so that merged branches (as metadata) are erased automatically, similar to how GitHub removes them after merging to master? Currently, there is no such option in a merge request to delete it after a successful merge or delete it after a successful merge to the default branch.

Hi @Barak Amar, Thanks for the detailed answers — that helped. I still have a couple of follow-up questions to make sure I understand the retention and cleanup behavior correctly. 1. Retention meaning in this config Given the configuration below:

{
  "branches": [
    {
      "branch_id": "main",
      "retention_days": 7
    }
  ],
  "default_retention_days": 1
}

What exactly happens after: • 1 day (for non-main branches)? • 7 days (for the main branch)? In other words, which objects are affected by retention (branch metadata, commits, underlying data/objects), and what’s the lifecycle/timeline for each in this setup? 2. Auto-delete branches that weren’t merged within 1 day My business requirement is: if a branch is not merged into master within 1 day, I want the branch and the underlying data it references to be deleted. • Is there a built-in way to enforce this policy in the system? • If not, is there an official job/tool/script you recommend for implementing this (e.g., something internal you mentioned exists), and where can I find it? • If there’s no supported job/tool, is the only option an application-level cleanup process (e.g., periodic automation calling the API to delete branches)?

All the objects uploaded and committed to lakeFS are referenced in commits. This data is kept and not deleted, as at any point in time you can reference or access this data. Even if you delete data and commit the deletion, we can still reference the old commit. The garbage collection process deletes the actual data of objects that were deleted from lakeFS, but to do that, we need to define which commits reference the data we want to access. Going over the commit log X days back based on the "retention_days" in the above configuration - these are the objects that we retain access to. All object data that is no longer accessed through these commits will be collected. Uncommitted - GC will not collect all data that is currently uncommitted on lakeFS. Currently, there is no branch merge policy as you described that lakeFS enforces. Suggest creating a feature request to capture your use-case.Here's the corrected text with explanations: A custom scheduled job or script using the lakeFS SDK would likely be the best approach to meet your business requirements.

Hi @Barak Amar, I deleted the branch(es) three days ago and expected the garbage collector to remove the underlying objects. However, I still see the data present after three days. How can I trigger the deletion of the data associated with those branches? According to the configuration, the default retention is 1 day.

Hi Dima, Garbage Collection happens for an object when all branches pointing to that object have their retention periods expired. I recommend going over the documentation here. lakeFS offers both a hosted solution which includes a managed garbage collection as well as an Enterprise on-prem solution which includes enhanced support. Feel free to explore these options