Hey! I am trying to run the Garbage collection using the command from the garbage collection tutorial (filled with my information)

spark-submit --class io.treeverse.gc.GarbageCollection \
    --packages org.apache.hadoop:hadoop-aws:2.7.7 \
    -c spark.hadoop.lakefs.api.url=<https://lakefs.example.com:8000/api/v1>  \
    -c spark.hadoop.lakefs.api.access_key=<LAKEFS_ACCESS_KEY> \
    -c spark.hadoop.lakefs.api.secret_key=<LAKEFS_SECRET_KEY> \
    -c spark.hadoop.fs.s3a.access.key=<S3_ACCESS_KEY> \
    -c spark.hadoop.fs.s3a.secret.key=<S3_SECRET_KEY> \
    <http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/lakefs-spark-client/0.16.0/lakefs-spark-client-assembly-0.16.0.jar> \
    example-repo us-east-1

And I got this error, any idea?

Exception in thread "main" java.lang.NullPointerException
	at io.treeverse.clients.ApiClient$$anon$7.get(ApiClient.scala:224)
	at io.treeverse.clients.ApiClient$$anon$7.get(ApiClient.scala:220)
	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243)
	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:193)
	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376)
	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112)
	at io.treeverse.clients.RequestRetryWrapper.wrapWithRetry(ApiClient.scala:325)
	at io.treeverse.clients.ApiClient.getBlockstoreType(ApiClient.scala:235)
	at io.treeverse.gc.GarbageCollection$.run(GarbageCollection.scala:150)
	at io.treeverse.gc.GarbageCollection$.main(GarbageCollection.scala:109)
	at io.treeverse.gc.GarbageCollection.main(GarbageCollection.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:966)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:191)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:214)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1054)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1063)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Spark version

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.2.4
      /_/

Using Scala version 2.12.15, OpenJDK 64-Bit Server VM, 11.0.29
Branch HEAD
Compiled by user centos on 2023-04-09T20:59:10Z
Revision 0ae10ac18298d1792828f1d59b652ef17462d76e
Url <https://github.com/apache/spark>
Type --help for more information.

LakeFS version 1.29.0 Thanks heart lakefs

It looks like the

NullPointerException

might be caused by a version mismatch between the hadoop libraries you are loading and the ones bundled with your spark installation. So maybe it's because you are using

hadoop-aws:2.7.7

but your spark version (3.2.4) ships with hadoop 3.2.x.

I will try it later and let you know! thanks

Hi @Carlos Luque - I’d be remiss if I didn’t point out that lakeFS Cloud includes managed GC, and the on prem Enterprise version has a simplified standalone GC.