Hi Guys! it’s been a while… 🙂 we’re trying to run the Garbage Collector and ran into some dependencies issues we tried running it (sweep stage - mark stage finished successfully) on spark 2.4.8 (2.4.7 with --packages does not work anymore because of bintray deprecations) when specifiying

hadoop-aws 2.7.7

in packages it installs a dependency of

aws-java-sdk 1.7.4

and we get an error for classDef not found (the class definition is found in newer

aws-java-sdk

but not in 1.7.4)

Caused by: java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSStaticCredentialsProvider
	at io.treeverse.clients.conditional.S3ClientBuilder$.build(S3ClientBuilder.scala:26)
	at io.treeverse.clients.BulkRemoverFactory$S3BulkRemover.getS3Client(BulkRemoverFactory.scala:69)
	at io.treeverse.clients.BulkRemoverFactory$S3BulkRemover.deleteObjects(BulkRemoverFactory.scala:58)

could you help us figure how to run the sweep? Thanks :)

Hey @Daniel Satubi! Sure 🙂 looping in @Tal Sofer

Thanks @Oz Katz! and hi @Daniel Satubi 🙂 looking into it

Thanks, I’ll add the full stack trace with the submit command

Thank you!

Thanks! 🙂 let me know if I can help test it somehow

Hi @Daniel Satubi 🙂 I’m getting back to you with updates. We ran some tests on the solution in the pr above and planning to run more validations. So far, things look good. so if everything works well we will release the fix by end of this week, hopefully before Thursday. I will of course continue to share updates with you

Thanks! 🙂 I read some of the discussion in the PR, are you testing with 2.4.7 or 2.4.8? which hadoop version in the spark jars?

i’m testing with 2.4.7, hadoop version 2.7.7

we have a 2.4.7 & 2.4.8 cluster with hadoop 2.7.3 - I’m not sure it’ll work I’ll see what we can do…

Hi @Daniel Satubi! i’m considering another optional solution that may be independent of your hadoop version. I will keep you updated.

Thanks! looking forward… 🥳

That’s perfect! 🤗 thanks for letting us know, and for offering to test things up! Sharing updates on our current status: during the last we tested the solution in https://github.com/treeverse/lakeFS/pull/4920, while it works with hadoop 2, it doesn’t work with hadoop 3 which broke our Spark3-hadoop3 client. We are now testing another solution that limits this change only to our hadoop2 builds. For that reason we will not be releasing the fix this week but during the next week after completing validations 🙂

Great! thank you very much 🙂 have a nice weekend

hi Daniel good morning!

Thanks! if you’d like to send us the artifact for testing before releasing we’d be more than happy 🙂

Thanks for the kind offer! I will be actually happy to do so. what email I can sent the artifact to?

daniels@windward.ai or even here if it works (not sure if there’s a size limit in slack)

thanks, will let you know once I sent it 🙂 in a meeting will do it right after

Thanks! Should I ran it with the same command as in the docs?

yes,

--packages org.apache.hadoop:hadoop-aws:2.7.7

it is

Thanks! will run an report back 🙂

Thanks for reporting back 🙂

It finished successfully but I don’t think the files we expected were deleted…`

previously we deleted all stale branches and run the mark job, we have a 300TB+ bucket for lakefs with about 20TB of actively accessed data, also we see the data is growing linearly so we assume 20TB is the size of the “main” give or take and the ~300TB is the data from the branches. we expected to see some changes after the sweep but none took effect… can we set the job to log more info?

Really happy that the artifact we shared with you fixes the dependency issue, thanks for testing it out!:heart_lakefs: As for your question, since it is related to the way GC works can I ask you to move it to our #help channel? I’m sure that other community members can benefit from it. When you move it there I would add more details about your GC setup. For example, what retention policy you have configured, how did you run the job (mark and then sweep or mark+sweep together), and attach logs you feel comfortable sharing. If you think there is a bug you are also welcome to open a github issue describing it.

Thanks! I’ll wait for the new run to finish and if there’s any problem I’ll write in the #help channel We’re very grateful for the quick replays and all the help 🙂

Our pleasure! thank you!

still running…I’ll report back tomorrow with all the results and our cluster versions

Cool! thank you!