Hi I`m trying to make the LakeFS work in databricks but enco

Question

Hi I`m trying to make the LakeFS work in databricks but encounter the error below ```Py4JJavaError An error occurred while calling o407 parquet java lang RuntimeException java lang ClassNotFoundException Class io lakefs LakeFSFileSystem not found``` I used the hadoop fs settings as mentioned in docus ` ```spark hadoop fs lakefs impl io lakefs LakeFSFileSystem``` Please pray any idea how to deal with that

Eden Ohana · Answer

Hi Zdenek Can you share your configuration

Zdenek Hruby · Answer

```sc jsc hadoopConfiguration set fs lakefs impl io lakefs LakeFSFileSystem sc jsc hadoopConfiguration set fs lakefs endpoint <http XX XXX XXX XXX 8000> sc jsc hadoopConfiguration set fs lakefs access key XXXXXXXXXXXXXXXXXXXX sc jsc hadoopConfiguration set fs lakefs secret key XXXXXXXXXXXXXXXXXXXXXXXXX sc jsc hadoopConfiguration set fs s3a access key XXXXXXXXXXXXXXXXXXXXXXXXX sc jsc hadoopConfiguration set fs s3a secret key XXXXXXXXXXXXXXXXXXXXXXXX ```

Zdenek Hruby · Answer

Hi Eden here is it The same result was with setting in cluster directly

Adi Polak · Answer

Hi Zdenek did you install the lakeFS file system library on the cluster This is a jar you download from the maven repository

Zdenek Hruby · Answer

Hi Adi yes I did

Zdenek Hruby · Answer

but I had to upload the jar file and then install it It was not possible per installer in UI man shrugging

Adi Polak · Answer

does your endpoint looks like this ```spark hadoop fs lakefs endpoint=<https lakefs example com api v1> ``` ^ could it be that you forgot the ` api v1`

Adi Polak · Answer

this is a very good <https lakefs io blog databricks lakefs integration tutorial |tutorial> it configures the cluster from the cluster UI yet might be very useful to you I would try removing the port 8000 from the endpoint as i don t recall it was necessary i might be wrong here

Zdenek Hruby · Answer

Cluster should work because lakefs client works fine I fixed the endpoint as you told me and now getting a different kind of error ```Py4JJavaError An error occurred while calling o408 parquet java lang RuntimeException unsupported URI scheme https lakeFS FileSystem currently supports translating s3 = gt s3a only```

Adi Polak · Answer

yay progress muscle what does the configuration look like now

Zdenek Hruby · Answer

``` secrets lakefs lakefs storage sk spark hadoop fs lakefs secret key XXXXX spark hadoop fs lakefs access key XXXXX spark hadoop fs lakefs endpoint spark hadoop fs lakefs impl io lakefs LakeFSFileSystem spark databricks delta preview enabled true``` this is from cluster configuration Account key works for the client and key and secret as well man shrugging

Adi Polak · Answer

```http gt https```

Adi Polak · Answer

also please remove the credentials slightly smiling face i mean from the slack

Zdenek Hruby · Answer

creadentials are fake just the format is same slightly smiling face But for sure I remove it

Adi Polak · Answer

oh so no worries

Adi Polak · Answer

i wonder if the https will fix it

Zdenek Hruby · Answer

no it doesn`t cry And the http get ``` message invalid API endpoint ```

Eden Ohana · Answer

What command are you running when getting the error

Zdenek Hruby · Answer

```spark read parquet <lakefs dbxdata main gendata parquet> ```

Eden Ohana · Answer

try to add to the configuration ```spark hadoop fs s3a access key spark hadoop fs s3a secret key```

Eden Ohana · Answer

the key and access key to your bucket

Zdenek Hruby · Answer

do I need this if I set <http spark hadoop fs azure account key lakefstest dfs core windows net|spark hadoop fs azure account key lakefstest dfs core windows net>

Oz Katz · Answer

< Zdenek Hruby> hi is this lakeFS installation using Azure Blob as the underlying object store currently the lakeFS HadoopFilesystem integration is only supported for AWS S3 based installations If you are on Azure you can use the yes on azure

Oz Katz · Answer

btw Azure aupport for the native HadoopFilesystem is on the <https docs lakefs io roadmap html hadoopfs support azure blob storage|roadmap>

Zdenek Hruby · Answer

Hi Oz Thanks for the explanation +1