Zdenek Hruby
01/16/2023, 4:15 PMPy4JJavaError: An error occurred while calling o407.parquet.
: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class io.lakefs.LakeFSFileSystem not found
I used the hadoop fs settings as mentioned in docus `
spark.hadoop.fs.lakefs.impl io.lakefs.LakeFSFileSystem
Please 🙏, any idea how to deal with that?Eden Ohana
01/16/2023, 4:25 PMZdenek Hruby
01/16/2023, 4:27 PMsc._jsc.hadoopConfiguration().set("fs.lakefs.impl", "io.lakefs.LakeFSFileSystem")
sc._jsc.hadoopConfiguration().set("fs.lakefs.endpoint", "<http://XX.XXX.XXX.XXX:8000>")
sc._jsc.hadoopConfiguration().set("fs.lakefs.access.key", "XXXXXXXXXXXXXXXXXXXX")
sc._jsc.hadoopConfiguration().set("fs.lakefs.secret.key", "XXXXXXXXXXXXXXXXXXXXXXXXX")
sc._jsc.hadoopConfiguration().set("fs.s3a.access.key", "XXXXXXXXXXXXXXXXXXXXXXXXX")
sc._jsc.hadoopConfiguration().set("fs.s3a.secret.key", "XXXXXXXXXXXXXXXXXXXXXXXX")
Adi Polak
01/16/2023, 4:34 PMZdenek Hruby
01/16/2023, 5:34 PMAdi Polak
01/16/2023, 6:19 PMspark.hadoop.fs.lakefs.endpoint=<https://lakefs.example.com/api/v1>
^ could it be that you forgot the /api/v1
Zdenek Hruby
01/16/2023, 6:41 PMPy4JJavaError: An error occurred while calling o408.parquet.
: java.lang.RuntimeException: unsupported URI scheme https, lakeFS FileSystem currently supports translating s3 => s3a only
Adi Polak
01/16/2023, 6:55 PMZdenek Hruby
01/16/2023, 7:05 PM<http://spark.hadoop.fs.azure.account.key.lakefstest.dfs.core.windows.net|spark.hadoop.fs.azure.account.key.lakefstest.dfs.core.windows.net> {{secrets/lakefs/lakefs-storage-sk}}
spark.hadoop.fs.lakefs.secret.key XXXXX
spark.hadoop.fs.lakefs.access.key XXXXX
spark.hadoop.fs.lakefs.endpoint <http://10.162.160.196:8000/api/v1>
spark.hadoop.fs.lakefs.impl io.lakefs.LakeFSFileSystem
spark.databricks.delta.preview.enabled true
this is from cluster configuration. Account key works for the client and key and secret as well 🤷♂️Adi Polak
01/16/2023, 7:07 PMhttp -> https
Zdenek Hruby
01/16/2023, 7:11 PMAdi Polak
01/16/2023, 7:11 PMZdenek Hruby
01/16/2023, 7:41 PM{
"message": "invalid API endpoint"
}
Eden Ohana
01/16/2023, 7:46 PMZdenek Hruby
01/16/2023, 7:47 PMspark.read.parquet("<lakefs://dbxdata/main/gendata.parquet>")
Eden Ohana
01/16/2023, 7:55 PMspark.hadoop.fs.s3a.access.key
spark.hadoop.fs.s3a.secret.key
Zdenek Hruby
01/16/2023, 8:14 PMOz Katz
01/16/2023, 8:35 PMZdenek Hruby
01/16/2023, 8:44 PM