Hello, are the garbage collection spark binaries all backwards compatible? I’m still running LakeFS server version 0.99.0 and I'd like to run committed and uncommitted GC individually. Should I used the spark client version that shipped with 0.99.0 or should I use a more recent version?

Hi @Jacob I think you might be conflating 2 different questions. lakeFS metadata client (GC client) is backwards compatible (newer GC clients support older lakeFS versions). However, BC is guaranteed for versions 1.x and not earlier. Regarding running committed and uncommitted GC separately - this is a functionality we no longer have. The GC client supports 2 modes (mark and/or sweep). It does this for both committed and uncommitted data. Separating committed and uncommitted GC is a functionality that no longer exists in newer versions.

Ah okay I see. Are there any concerns with running the uncommitted and committed GC operations that existed at the

v0.99.0

release? I saw that one of the markdown files that that uncommitted GC was still in a beta mode at that commit but I think that might have just been a stale warning

lakeFS metadata client and lakeFS server have different version cadence. Can you be more specific about which server/client versions you want to run?

This is the guide I’m following https://github.com/treeverse/lakeFS/blob/v0.99.0/docs/howto/garbage-collection-uncommitted.md

If your using the client version specified in this document and your lakeFS server version is 0.99.0 you should be good

Okay thanks. Is it safe to run the uncommitted garbage collector from this point in time or did it have bugs that were fixed later on?

Changelog for both client and server is available and you can scan through it and identify any changes related to GC. This version is too old for me to pin point something specific. I can, with high certainty, guarantee there have been bug fixes in GC since this version. However none of which related to data integrity

Hi @Jacob, This version is over 3 years old, we had multiple securities fixes over the years, I strongly recommend upgrading ASAP.

Thanks Iddo. Are you referring to the garbage collector or the server itself

The lakeFS Server. (Although very reasonable there have been updates on the GC as well)

Is the backend and the GC JAR decoupled?

They are mostly decoupled - you can run a newer GC with older data. As @Iddo Avneri mentioned, I would probably upgrade lakeFS first, run GC later. But TBH this is a BIG version jump, and from a version (0.99) that had no compatibility guarantees. If I were doing this, I would scan the entire CHANGELOG between these versions. 1.0 was a major release (the first), and anything can break between <1.0.0 and >=1.0.0. We even removed a bunch of features that turned out to be duds.