Hello! I'm trying to query data from lakeFS, run locally, using pyspark, however, I get "java.lang.RuntimeException: java.lang.ClassNotFoundException: Class io.lakefs.LakeFSFileSystem not found". I am running a spark shell with --packages io.lakefshadoop lakefs assembly0.1.12 as per the doc https://docs.lakefs.io/integrations/spark.html. Could someone explain why LakeFSFileSystem might not be found?

Hey @Asher Song, just to rule out any maven issues, could you try replacing the

--packages <...>

flag with:

// wrong jar was here

I still seem to get the same error

I have't tried building the jar, but i have been able to copy assembled jar to jars/ folder to access lakefs paths from spark. documentation here : https://docs.lakefs.io/reference/spark-client.html

@Chandra Akkinepalli note that this doc refers to the lakeFS metadata client, which allows listing files from lakeFS but not reading file contents

@Yoni Augarten, thanks for mentioning that.

Sure 😊

@Yoni Augarten, i am able to run something like this

spark-submit spark_changes.py --jars <http://treeverse-clients-us-east.s3-website-us-east-1.amazonaws.com/hadoop/hadoop-lakefs-assembly-0.1.12.jar>

and read from lakefs, i am able to follow the instructions on https://docs.lakefs.io/integrations/spark.html , created hdfs-site.xml to have appropriate settings.

@Yoni Augarten I’m only using a single --packages flag and I’m still getting the same error.

Hi, Sorry to hear you've some difficulties. Right now lakeFSFS is only supported when lakeFS uses an s3 or compatible backing store: lakeFSFS itself will directly access that having store. What backing store do you have? We're actively seeking people to run an upcoming beta (don't tell anyone... 😇) Also, it's a holiday over here, so I apologize in advance for lengthier than usual response times.

Oh I see. Currently my data is stored locally on my machine

Local data storage will not work with a distributed system like Spark, at least not without forcing you to do a significant amount of configuration work. You might look at the lakeFS "everything bagel", or playground account . Again, sorry but I don't expect us to provide good response times for the upcoming day.

Another possibility (if your situation allows it) is to use AWS (or any other cloud provider with an S3-compatible object store, which is more or less all of them) and use the free tier for evaluation purposes

You can also consider a local installation of min.io