I'm getting ```java.lang.NoSuchMethodException: co...
# dev
a
I'm getting
Copy code
java.lang.NoSuchMethodException: com.databricks.spark.metrics.FileSystemWithMetrics.getAmazonS3Client()
when writing using HadoopFS (
io.lakefs:hadoop-lakefs-assembly:0.1.4
). Has anyone seen this?
@Barak Amar this seems to be caused by MetadataClient, which tries to extract the method to grab metrics. That fails to catch
NoSuchMethodError
on the assembled client, probably because of some DataBricks weirdness that causes us to have the wrong exception class there. Is there any way for me to override these metrics?
b
At the time we wrote lakeFSFS we assumed we will use the AWS s3 client in order to grab the underlying etag. When we FileSystem returned by the API doesn't include this runtime information, we try to resolve it by calling the storage using the s3 client.
a
We'll run into this at other places, too.
Although only when running with an old lakeFS on DataBricks, right?
b
1. When the FileStatus underlying implementation doesn't hold ETag property. 2. When the underlying s3 client implementation is not AWS S3.