Hi I am trying to run garbage collector following the docs i

Question

Hi I am trying to run garbage collector following the docs instructions and unfortunately I cannot progress Please note that I am not a Scala developer so it is very likely that I made some stupid mistakes in my setup Kindly appreciate any help raised hands I am using Spark 3 0 1 and I installed the following dependencies ``` ``` I am running following command ```spark submit class io treeverse clients GarbageCollector packages org apache hadoop hadoop aws 3 3 1 c spark hadoop lakefs api url= https lt my lakefs api url gt api v1 c spark hadoop lakefs api access key= lt my lfs access key gt c spark hadoop lakefs api secret key= lt my lfs secret key gt c spark hadoop fs s3a access key= lt my s3 access key gt c spark hadoop fs s3a secret key= lt my s3 secret key gt $SPARK HOME jars lakefs spark client 301 2 12 0 5 1 jar lt my repo gt eu west 1``` and I get the following error ```Exception in thread main com google common util concurrent ExecutionError java lang NoClassDefFoundError dev failsafe function CheckedSupplier at com google common cache LocalCache$Segment get LocalCache java 2261 at com google common cache LocalCache get LocalCache java 4000 at com google common cache LocalCache$LocalManualCache get LocalCache java 4789 at io treeverse clients ApiClient$ get ApiClient scala 43 at io treeverse clients GarbageCollector$ main GarbageCollector scala 273 at io treeverse clients GarbageCollector main GarbageCollector scala at java base jdk internal reflect NativeMethodAccessorImpl invoke0 Native Method at java base jdk internal reflect NativeMethodAccessorImpl invoke NativeMethodAccessorImpl java 62 at java base jdk internal reflect DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl java 43 at java base java lang reflect Method invoke Method java 566 at org apache spark deploy JavaMainApplication start SparkApplication scala 52 at $apache$spark$deploy$SparkSubmit$$runMain SparkSubmit scala 928 at org apache spark deploy SparkSubmit doRunMain$1 SparkSubmit scala 180 at org apache spark deploy SparkSubmit submit SparkSubmit scala 203 at org apache spark deploy SparkSubmit doSubmit SparkSubmit scala 90 at org apache spark deploy SparkSubmit$$anon$2 doSubmit SparkSubmit scala 1007 at org apache spark deploy SparkSubmit$ main SparkSubmit scala 1016 at org apache spark deploy SparkSubmit main SparkSubmit scala ```

Cristian Caloian · Answer

Does anyone have a Docker image with all the setup for running GC

Idan Novogroder · Answer

Hi Cristian I m trying to look at the logs and find the issue Hope to come up with a good answer by the end of the day

Yoni Augarten · Answer

Hey < Cristian Caloian> transitive dependencies in the world of Hadoop can prove tricky I suggest using our uber jar which contains all the dependencies required for garbage collection You can find it in our public S3 bucket here ``````

Jonathan Rosenberg · Answer

Hi < Cristian Caloian> slightly smiling face Can you list the Hadoop dependencies you have in your Spark jars directory `$SPARK HOME jars` We can start with validating that the `hadoop aws` version matches the hadoop versions in your `jars` dir

Cristian Caloian · Answer

Thank you all for you replies raised hands Below are all jars that are hadoop related in my `$SPARK HOME jars` ```avro mapred 1 8 2 hadoop2 jar dropwizard metrics hadoop metrics2 reporter 0 1 2 jar hadoop annotations 3 2 0 jar hadoop auth 3 2 0 jar hadoop aws 3 3 1 jar hadoop client 3 2 0 jar hadoop common 3 2 0 jar hadoop hdfs client 3 2 0 jar hadoop lakefs assembly 0 1 8 jar hadoop mapreduce client common 3 2 0 jar hadoop mapreduce client core 3 2 0 jar hadoop mapreduce client jobclient 3 2 0 jar hadoop yarn api 3 2 0 jar hadoop yarn client 3 2 0 jar hadoop yarn common 3 2 0 jar hadoop yarn registry 3 2 0 jar hadoop yarn server common 3 2 0 jar hadoop yarn server web proxy 3 2 0 jar parquet hadoop 1 10 1 jar``` also all lakefs realted jars ```hadoop lakefs assembly 0 1 8 jar lakefs spark client 301 2 12 0 5 1 jar```

Cristian Caloian · Answer

I will try now with the uber jar suggested by < Yoni Augarten>

Cristian Caloian · Answer

I tried the `spark submit` command above but using the uber jar instead I am getting the same error This might be completely nonsense but when I look at the <https github com treeverse lakeFS blob master clients spark core src main scala io treeverse clients GarbageCollector scala|GarbageCollector scala> file trying to run all imports I noticed `import org json4s native JsonMethods ` fails ```scala gt import org json4s native JsonMethods lt console gt 23 error object native is not a member of package org json4s import org json4s native JsonMethods ```

Tal Sofer · Answer

Thanks < Cristian Caloian> for using the uber jar and sharing the dependency list Can you please try to also change this line ` packages org apache hadoop hadoop aws 3 3 1` into ` packages org apache hadoop hadoop aws 3 2 0` Your spark submit command will look as follows ```spark submit class io treeverse clients GarbageCollector packages org apache hadoop hadoop aws 3 2 0 c spark hadoop lakefs api url= https lt my lakefs api url gt api v1 c spark hadoop lakefs api access key= lt my lfs access key gt c spark hadoop lakefs api secret key= lt my lfs secret key gt c spark hadoop fs s3a access key= lt my s3 access key gt c spark hadoop fs s3a secret key= lt my s3 secret key gt PATH TO UBER JAR lt my repo gt eu west 1``` And when you say that you are getting the same error are you getting the exact stack trace or the same exception if you can share the stacktrace it will be great slightly smiling face

Cristian Caloian · Answer

Hi < Tal Sofer> thank you for the help I tried your suggestion but I get the same exception and stack trace See the long output below ``` base jovyan lfs client test 0 ~$ spark submit class io treeverse clients GarbageCollector gt packages org apache hadoop hadoop aws 3 2 0 gt c spark hadoop lakefs api url= https lt my lakefs api url gt api v1 gt c spark hadoop lakefs api access key= lt my lfs access key gt gt c spark hadoop lakefs api secret key= lt my lfs secret key gt gt c spark hadoop fs s3a access key= lt my s3 access key gt gt c spark hadoop fs s3a secret key= lt my s3 secret key gt gt c spark executor extraLibraryPath= home jovyan jars lakefs spark client 301 assembly 0 5 1 jar gt c spark executor extraLibraryPath= home jovyan jars hadoop aws 3 2 0 jar gt home jovyan jars lakefs spark client 301 assembly 0 5 1 jar gt lt my repo gt eu west 1 WARNING An illegal reflective access operation has occurred WARNING Illegal reflective access by org apache spark unsafe Platform file usr local spark 3 0 1 bin hadoop3 2 jars spark unsafe 2 12 3 0 1 jar to constructor java nio DirectByteBuffer long int WARNING Please consider reporting this to the maintainers of org apache spark unsafe Platform WARNING Use illegal access=warn to enable warnings of further illegal reflective access operations WARNING All illegal access operations will be denied in a future release Ivy Default Cache set to home jovyan ivy2 cache The jars for the packages stored in home jovyan ivy2 jars loading settings url = jar file usr local spark 3 0 1 bin hadoop3 2 jars ivy 2 4 0 jar org apache ivy core settings ivysettings xml org apache hadoop hadoop aws added as a dependency resolving dependencies org apache spark spark submit parent d996c5e0 6b56 43e8 8eff 3c80a1970dd0 1 0 confs default found org apache hadoop hadoop aws 3 2 0 in central found com amazonaws aws java sdk bundle 1 11 375 in central resolution report resolve 139ms artifacts dl 3ms modules in use com amazonaws aws java sdk bundle 1 11 375 from central in default org apache hadoop hadoop aws 3 2 0 from central in default | | modules || artifacts | | conf | number| search|dwnlded|evicted|| number|dwnlded| | default | 2 | 0 | 0 | 0 || 2 | 0 | retrieving org apache spark spark submit parent d996c5e0 6b56 43e8 8eff 3c80a1970dd0 confs default 0 artifacts copied 2 already retrieved 0kB 4ms 22 10 31 08 53 13 WARN NativeCodeLoader Unable to load native hadoop library for your platform using builtin java classes where applicable Using Spark s default log4j profile org apache spark log4j defaults properties 22 10 31 08 53 13 INFO SparkContext Running Spark version 3 0 1 22 10 31 08 53 13 INFO ResourceUtils ============================================================== 22 10 31 08 53 13 INFO ResourceUtils Resources for spark driver 22 10 31 08 53 13 INFO ResourceUtils ============================================================== 22 10 31 08 53 13 INFO SparkContext Submitted application GarbageCollector 22 10 31 08 53 13 INFO SecurityManager Changing view acls to jovyan 22 10 31 08 53 13 INFO SecurityManager Changing modify acls to jovyan 22 10 31 08 53 13 INFO SecurityManager Changing view acls groups to 22 10 31 08 53 13 INFO SecurityManager Changing modify acls groups to 22 10 31 08 53 13 INFO SecurityManager SecurityManager authentication disabled ui acls disabled users with view permissions Set jovyan groups with view permissions Set users with modify permissions Set jovyan groups with modify permissions Set 22 10 31 08 53 14 INFO Utils Successfully started service sparkDriver on port 40379 22 10 31 08 53 14 INFO SparkEnv Registering MapOutputTracker 22 10 31 08 53 14 INFO SparkEnv Registering BlockManagerMaster 22 10 31 08 53 14 INFO BlockManagerMasterEndpoint Using org apache spark storage DefaultTopologyMapper for getting topology information 22 10 31 08 53 14 INFO BlockManagerMasterEndpoint BlockManagerMasterEndpoint up 22 10 31 08 53 14 INFO SparkEnv Registering BlockManagerMasterHeartbeat 22 10 31 08 53 14 INFO DiskBlockManager Created local directory at tmp blockmgr 54820e8e 476c 420d 873a 00c02cfac502 22 10 31 08 53 14 INFO MemoryStore MemoryStore started with capacity 434 4 MiB 22 10 31 08 53 14 INFO SparkEnv Registering OutputCommitCoordinator 22 10 31 08 53 14 INFO Utils Successfully started service SparkUI on port 4040 22 10 31 08 53 14 INFO SparkUI Bound SparkUI to 0 0 0 0 and started at 22 10 31 08 53 14 INFO SparkContext Added JAR file home jovyan ivy2 jars org apache hadoop hadoop aws 3 2 0 jar at with timestamp 1667206394368 22 10 31 08 53 14 INFO SparkContext Added JAR file home jovyan ivy2 jars com amazonaws aws java sdk bundle 1 11 375 jar at with timestamp 1667206394368 22 10 31 08 53 14 INFO SparkContext Added JAR file home jovyan jars lakefs spark client 301 assembly 0 5 1 jar at with timestamp 1667206394369 22 10 31 08 53 14 INFO Executor Starting executor ID driver on host lfs client test 0 22 10 31 08 53 14 INFO Utils Successfully started service org apache spark network netty NettyBlockTransferService on port 34953 22 10 31 08 53 14 INFO NettyBlockTransferService Server created on lfs client test 0 34953 22 10 31 08 53 14 INFO BlockManager Using org apache spark storage RandomBlockReplicationPolicy for block replication policy 22 10 31 08 53 14 INFO BlockManagerMaster Registering BlockManager BlockManagerId driver lfs client test 0 34953 None 22 10 31 08 53 14 INFO BlockManagerMasterEndpoint Registering block manager lfs client test 0 34953 with 434 4 MiB RAM BlockManagerId driver lfs client test 0 34953 None 22 10 31 08 53 14 INFO BlockManagerMaster Registered BlockManager BlockManagerId driver lfs client test 0 34953 None 22 10 31 08 53 14 INFO BlockManager Initialized BlockManager BlockManagerId driver lfs client test 0 34953 None Exception in thread main com google common util concurrent ExecutionError java lang NoClassDefFoundError dev failsafe function CheckedSupplier at com google common cache LocalCache$Segment get LocalCache java 2261 at com google common cache LocalCache get LocalCache java 4000 at com google common cache LocalCache$LocalManualCache get LocalCache java 4789 at io treeverse clients ApiClient$ get ApiClient scala 43 at io treeverse clients GarbageCollector$ main GarbageCollector scala 273 at io treeverse clients GarbageCollector main GarbageCollector scala at java base jdk internal reflect NativeMethodAccessorImpl invoke0 Native Method at java base jdk internal reflect NativeMethodAccessorImpl invoke NativeMethodAccessorImpl java 62 at java base jdk internal reflect DelegatingMethodAccessorImpl invoke DelegatingMethodAccessorImpl java 43 at java base java lang reflect Method invoke Method java 566 at org apache spark deploy JavaMainApplication start SparkApplication scala 52 at $apache$spark$deploy$SparkSubmit$$runMain SparkSubmit scala 928 at org apache spark deploy SparkSubmit doRunMain$1 SparkSubmit scala 180 at org apache spark deploy SparkSubmit submit SparkSubmit scala 203 at org apache spark deploy SparkSubmit doSubmit SparkSubmit scala 90 at org apache spark deploy SparkSubmit$$anon$2 doSubmit SparkSubmit scala 1007 at org apache spark deploy SparkSubmit$ main SparkSubmit scala 1016 at org apache spark deploy SparkSubmit main SparkSubmit scala Caused by java lang NoClassDefFoundError dev failsafe function CheckedSupplier at io treeverse clients ApiClient$$anon$1 call ApiClient scala 44 at io treeverse clients ApiClient$$anon$1 call ApiClient scala 43 at com google common cache LocalCache$LocalManualCache$1 load LocalCache java 4792 at com google common cache LocalCache$LoadingValueReference loadFuture LocalCache java 3599 at com google common cache LocalCache$Segment loadSync LocalCache java 2379 at com google common cache LocalCache$Segment lockedGetOrLoad LocalCache java 2342 at com google common cache LocalCache$Segment get LocalCache java 2257 17 more Caused by java lang ClassNotFoundException dev failsafe function CheckedSupplier at java base jdk internal loader BuiltinClassLoader loadClass BuiltinClassLoader java 581 at java base jdk internal loader ClassLoaders$AppClassLoader loadClass ClassLoaders java 178 at java base java lang ClassLoader loadClass ClassLoader java 522 24 more 22 10 31 08 53 14 INFO SparkContext Invoking stop from shutdown hook 22 10 31 08 53 14 INFO SparkUI Stopped Spark web UI at 22 10 31 08 53 14 INFO MapOutputTrackerMasterEndpoint MapOutputTrackerMasterEndpoint stopped 22 10 31 08 53 14 INFO MemoryStore MemoryStore cleared 22 10 31 08 53 14 INFO BlockManager BlockManager stopped 22 10 31 08 53 14 INFO BlockManagerMaster BlockManagerMaster stopped 22 10 31 08 53 14 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint OutputCommitCoordinator stopped 22 10 31 08 53 14 INFO SparkContext Successfully stopped SparkContext 22 10 31 08 53 14 INFO ShutdownHookManager Shutdown hook called 22 10 31 08 53 14 INFO ShutdownHookManager Deleting directory tmp spark 68224eda 9da1 4aa4 b316 dec0b0003d9d 22 10 31 08 53 14 INFO ShutdownHookManager Deleting directory tmp spark fc8a6690 aa83 4347 b3da 7cc9d048b591```

Or Tzabary · Answer

hey < Cristian Caloian> thanks for the information we ll take a look and update

Jonathan Rosenberg · Answer

Hi < Cristian Caloian> This looks like a good ol dependency fun To make things a bit cleaner can you delete the `lakefs spark client 301 2 12 0 5 1 jar` from the `$SPARK HOME jars` directory and run it again only with the assembly jar the assembly jar contains a shaded version of `com google common ` which means that you probably ran the non assembly jar

Cristian Caloian · Answer

Thanks for the suggestion < Jonathan Rosenberg> I get a different error now which I take as we made some progress Please see below the output this time as a file so it can be collapsed

Cristian Caloian · Answer

I now have in my `$SPARK HOME jars` both ``` base jovyan lfs client test 1 0 ~$ ls $SPARK HOME jars | grep lakefs hadoop lakefs assembly 0 1 8 jar lakefs spark client 301 assembly 0 5 1 jar``` Is that the correct setup Or only `lakefs spark client 301 assembly 0 5 1 jar` should be enough

Jonathan Rosenberg · Answer

You can remove `hadoop lakefs assembly 0 1 8 jar` It s not necessary for the GC A question did you create a lakeFS user other than the admin and are using its credentials as the lakeFS key and secret input for the GC process Also it would be very helpful if you ll attach the logs from the lakeFS server after running GC Thanks

Cristian Caloian · Answer

gt You can remove `hadoop lakefs assembly 0 1 8 jar` It s not necessary for the GC Got it Would I need it if I want to use the Spark client to work with files on LakeFS gt did you create a lakeFS user other than the admin and are using its credentials as the lakeFS key and secret input for the GC process I am using my user credentials which also has admin rights It also has policies attached for the repo I am trying to run the GC gt it would be very helpful if you ll attach the logs from the lakeFS server after running GC Working on it Let me get back to you later

Jonathan Rosenberg · Answer

gt Got it Would I need it if I want to use the Spark client to work with files on LakeFS You can work with Spark and lakeFS without the lakeFS Hadoop Filesystem jar using lakeFS s but from performance POV it would be better if you ve used the gt I am using my user credentials which also has admin rights It also has policies attached for the repo I am trying to run the GC gt Working on it Let me get back to you later Great

Cristian Caloian · Answer

Hi Sorry for the late reply By the trial and error principle I think I managed to get a working version At least no errors in the output There are a couple of things I would like to run by you just to double check if everything looks good I am pasting attaching below the dependencies the command the output and the warning log messages I used Spark 3 0 1 with Hadoop 3 2 downloaded from here ```<https archive apache org dist spark spark 3 0 1 spark 3 0 1 bin hadoop3 2 tgz>``` Regarding the dependencies do you have an explanation why both of these are needed It actually doesn t work otherwise Not that it bothers me but it looks a bit strange especially the versions 301 vs 312 I have attached the full list ```hadoop lakefs assembly 0 1 8 jar lakefs spark client 301 assembly 0 5 1 jar lakefs spark client 312 hadoop3 assembly 0 5 1 jar``` The command to run the GC ```spark submit class io treeverse clients GarbageCollector c spark hadoop lakefs api url= c spark hadoop lakefs api access key= c spark hadoop lakefs api secret key= c spark hadoop fs s3a access key= c spark hadoop fs s3a secret key= $SPARK HOME jars lakefs spark client 312 hadoop3 assembly 0 5 1 jar lt repo name gt eu west 1``` The output stdout not much info here because I think the GC policy wasn t set up properly but I will test different configurations in the next days ```gcCommitsLocation s3a lt bucket name gt data lakefs retention gc commits run id=981b2004 d427 4038 8d2f 20f100f73eab commits csv gcAddressesLocation s3a lt bucket name gt data lakefs retention gc addresses apiURL https lt lakefs url gt api v1 getAddressesToDelete use 50 partitions for ranges getAddressesToDelete use 200 partitions for addresses Expired addresses + + + |address|mark id| + + + + + + addressDFLocation s3a lt bucket name gt data lakefs retention gc addresses ``` And the logs stderr very long file with about 13k lines there are no errors but I want to share the warnings just to check with you if there is anything I need to take care ```WARNING An illegal reflective access operation has occurred WARNING Illegal reflective access by org apache spark unsafe Platform file usr local spark 3 0 1 bin hadoop3 2 jars spark unsafe 2 12 3 0 1 jar to constructor java nio DirectByteBuffer long int WARNING Please consider reporting this to the maintainers of org apache spark unsafe Platform WARNING Use illegal access=warn to enable warnings of further illegal reflective access operations WARNING All illegal access operations will be denied in a future release 22 11 03 17 05 57 WARN NativeCodeLoader Unable to load native hadoop library for your platform using builtin java classes where applicable 22 11 03 17 06 00 WARN MetricsConfig Cannot locate configuration tried hadoop metrics2 s3a file system properties hadoop metrics2 properties 22 11 03 17 06 14 WARN AbstractS3ACommitterFactory Using standard FileOutputCommitter to commit work This is slow and potentially unsafe 22 11 03 17 06 15 WARN AbstractS3ACommitterFactory Using standard FileOutputCommitter to commit work This is slow and potentially unsafe ```

Cristian Caloian · Answer

gt You can work with Spark and lakeFS without the lakeFS Hadoop Filesystem jar using lakeFS s but from performance POV it would be better if you ve used the I have already set up the S3 gateway but I thought it s better to have the Hadoop Filesystem set up and create some boilerplate for our users

Or Tzabary · Answer

< Cristian Caloian> thanks for the information we ll have a look and get back to you

Cristian Caloian · Answer

Regarding the Hadoop Filesystem Spark lakefs client I m trying to use the same Docker image to load some data and it seems that it works i e I can read a csv file in a dataframe and run `df show ` However when I start the `spark shell` I get the following error message ``` problems summary ERRORS Server access error at url <https dl bintray com spark packages maven io lakefs lakefs parent 0 lakefs parent 0 jar|https dl bintray com spark packages maven io lakefs lakefs parent 0 lakefs parent 0 jar> javax net ssl SSLHandshakeException Remote host terminated the handshake ``` And when I m trying to read the file I get ```scala gt val df = spark read option delimiter option header true csv dataPath 22 11 04 10 48 19 WARN MetricsConfig Cannot locate configuration tried hadoop metrics2 s3a file system properties hadoop metrics2 properties df org apache spark sql DataFrame = sepal length string sepal width string 3 more fields ``` Again it seems that the data is successfully read ```scala gt df show + + + + + + |sepal length|sepal width|petal length|petal width|variety| + + + + + + | 5 1| 3 5| 1 4| 2| Setosa| | 4 9| 3| 1 4| 2| Setosa| ``` Do you have a suggestion on how to fix this The error and the warning that is

Or Tzabary · Answer

< Cristian Caloian> it seems like an issue with bintray s serving the file

Or Tzabary · Answer

<http dl bintray com|dl bintray com> unexpectedly closed the connection

Or Tzabary · Answer

which docker image are you using

Or Tzabary · Answer

and can you share the spark shell command you re running

Cristian Caloian · Answer

This is the command I use the start the shell ```spark shell conf spark hadoop fs s3a access key= = conf spark hadoop fs s3a secret key= = conf spark hadoop fs s3a endpoint= <https s3 eu west 1 amazonaws com> conf spark hadoop fs lakefs impl= io lakefs LakeFSFileSystem conf spark hadoop fs lakefs access key= = conf spark hadoop fs lakefs secret key= = conf spark hadoop fs lakefs endpoint= = packages io lakefs hadoop lakefs assembly 0 1 8``` For the image I am using an existing jupyter image just so I can experiment quickly and because it had Spark installed but I will create separate images one for running the GC and one for the LakeFS Spark client Do you have a suggestion on how I can fix the bintray issue Do I need to install some dependency I sincerely apologize for all these questions but I am new to the Spark Scala world

Or Tzabary · Answer

Nothing to be sorry about we re happy to have you here I ll try to reproduce it soon and see if I can provide a quick mitigation

Cristian Caloian · Answer

Thank you < Or Tzabary> I just wanted to clarify something about the image I created it with the dependencies I mentioned

Or Tzabary · Answer

fyi bintray closed their services a year and a half ago maybe the jupyter image you re using is configured to use bintray as its maven repository

Or Tzabary · Answer

trying using the latest jupyter version to see if it helps

Or Tzabary · Answer

I managed to run the `spark shell` command you sent without any errors

Or Tzabary · Answer

please let me know if it helps

Cristian Caloian · Answer

Then it is probably from my end I will try to create a slimmer image and re run the steps and let you know how it goes

Tal Sofer · Answer

Hi < Cristian Caloian> happy that things are working for you I have one clarification question when you say gt Regarding the dependencies do you have an explanation why both of these are needed It actually doesn t work otherwise Not that it bothers me but it looks a bit strange especially the versions 301 vs 312 I have attached the full list gt ```hadoop lakefs assembly 0 1 8 jar gt lakefs spark client 301 assembly 0 5 1 jar gt lakefs spark client 312 hadoop3 assembly 0 5 1 jar``` What do you mean do you mean that you must have both ```lakefs spark client 301 assembly 0 5 1 jar lakefs spark client 312 hadoop3 assembly 0 5 1 jar``` In your jars folder for GC to work

Cristian Caloian · Answer

Hi < Tal Sofer> exactly I tried removing `lakefs spark client 301 assembly 0 5 1 jar` as I felt like it must be redundant but it didn t work after I then thought it must have to do with the fact that I am using Spark 3 0 1 But then I tried with Spark 3 1 2 and only `lakefs spark client 312 hadoop3 assembly 0 5 1 jar` but still not working Let me double check just to be 100 Do you have a recommendation for what versions Spark and jar should I be using

Tal Sofer · Answer

Thanks < Cristian Caloian> `lakefs spark client 301 assembly 0 5 1 jar` indeed should be redundant If you share with us the error you are getting as a result of removing it we will be happy to help troubleshooting gt have a recommendation for what versions Spark and jar should I be using Since your are using hadoop 3 I would recommend using this version of the jar `lakefs spark client 312 hadoop3 assembly 0 5 1 jar` This version is validated with Spark 3 1 2 but it may work with other Spark versions as I think it worked for you Running it with Spark 3 1 2 would be best if this work for you slightly smiling face

Cristian Caloian · Answer

< Tal Sofer> it is actually working as expected Spark 3 1 2 Hadoop 3 2 and the following lakefs jars ```hadoop lakefs assembly 0 1 8 jar lakefs spark client 312 hadoop3 assembly 0 5 1 jar``` I must ve gotten confused while trying various versions combinations Sorry for the false alarm and thank you for the support pray With the dependencies nailed down I now need to figure out how to create an image that I can use for running a Kubernetes job of kind `SparkApplication` I might need to ask more questions here if I run into trouble

Tal Sofer · Answer

Great news Our pleasure we are here to support you

Tal Sofer · Answer

Mind sharing your use case and infra Also are you locked to k8s to run your Spark apps there

Cristian Caloian · Answer

Yes pretty much all our services run on Kubernetes and I think it would be good to have a recurrent Spark job running the garbage collection on LakeFS for our users Do you have a suggestion or alternative solution

Tal Sofer · Answer

Thanks for sharing the details Where is your infra running k8s is a valid option although you will have to make sure your cluster has sufficient resources to run against your env If this is an option for you easiest may be to use a managed Spark env like databricks or EMR