Hi, Could someone help me? When I try to read csv file from lakeFS using Apache Spark, I get the following exception?

Caused by: io.lakefs.clients.api.ApiException: Content type "text/html; charset=utf-8" is not supported for type: class io.lakefs.clients.api.model.ObjectStats
  at io.lakefs.clients.api.ApiClient.deserialize(ApiClient.java:820)
  at io.lakefs.clients.api.ApiClient.handleResponse(ApiClient.java:1018)
  at io.lakefs.clients.api.ApiClient.execute(ApiClient.java:942)
  at io.lakefs.clients.api.ObjectsApi.statObjectWithHttpInfo(ObjectsApi.java:951)
  at io.lakefs.clients.api.ObjectsApi.statObject(ObjectsApi.java:926)
  at io.lakefs.LakeFSFileSystem.getFileStatus(LakeFSFileSystem.java:534)

Do any body face this issue before?

Hi, can you share the spark configuration you set - it looks like the lakefs api endpoint is not set right

LakeFS has been deployed in docker

can you set it to http://lakefs:8000/api/v1

Looks like it helped but now I have another exception. Does it mean I have incorrect access key and secret key?

yes, you need to use the lakefs key/secret

yep, It was a mistake in keys. I made a next step but still have exception:

are you using lakefs with 'local' adapter?

Hey @Artsiom Yudovin (cc @Barak Amar) - indeed the native hadoop FS used with Spark only supports s3a. I've opened an issue to provide better error messages when using an unsupported storage backend. To access data stored in lakeFS when running with locally stored data, you can directly use S3A to talk to lakeFS using the lakeFS S3 Gateway

thank you for your help! I started to use s3a but I still have issue. I’m not sure that this issue is connected to lakefs. It looks like I need to configure AmazonS3Client to work with docekr service name.

hmm. The

s3a.endpoint

should be the same as configured for gateways.s3.domain_name in the lakeFS configuration.

yes

I'm not 100% how to setup the DNS for this to work between containers, but generally lakeFS differentiates S3 gateway requests from other API requests based on the hostname used, this is why the endpoint needs to match up with the config. If alright with you I'd like to advise with people on the team to see what's the recommended way of doing this with a Docker setup. Let me do my research and get back to you around noon tomorrow (I'm GMT+3, so it's currently night time here).

yeah. thank you 🙏 ! FYI spark version is 3.0.2, hadoop-aws is 2.7.4

Thanks! that's helpful. Will update here tomorrow then.

JFYI: I changed the docker service name to IP and begin to get the following exception. But I have this bucket. No need to answer now. it’s ok to wait your response tomorrow.

Hey @Artsiom Yudovin! After digging a bit into this, the best solution I've found is to use docker-compose'

links

to match up the value given in the lakeFS config for

LAKEFS_GATEWAYS_S3_DOMAIN_NAME

with the link name. I've uploaded a detailed working example here - it should be usable as is, but probably easy to adapt to your existing configuration. It's configured to run Spark 3.0, so I believe it should work for you. Let me know how this works!

thank you! small question: Do I need to have S3 under lakefs? In case of I want to deploy it locally without using any tool from cloud. Do I have such kind of opportunity?

you have to use the same hostname as given in

LAKEFS_GATEWAYS_S3_DOMAIN_NAME

- the S3 protocol doesn't work well with IP addresses

got u, thank you! I will try to use example and come back with feedback

I've updated the example to use a local storage adapter - this means the data itself will be stored inside the docker container, so not really suitable for production (but faster and easier for local testing)

So, I try to use your example and have some trouble with spark execution. Spark doesn’t execute any stage and wait something. In spite of this, I try to execute spark not from docker. I execute from my machine and get the following exception:

hmm, for this config to work, you’d need to set

LAKEFS_GATEWAYS_S3_DOMAIN_NAME

to s3.local.lakefs.io - is that properly configured?

Do you mean configured for my machine?

configured for the lakefs container

I used container from this example

following the convention documented here: https://docs.lakefs.io/reference/configuration.html#using-environment-variables

Do I really need to use it if I deploy lakefs as docker container ?

you must tell lakefs which hostname will be used to accept s3 requests. it has to match the hostname you’re using as s3a.endpoint otherwise lakefs will not be able to parse spark’s requests

ah, sorry, my fail, I didn’t get that in example we have s3.docker.lakefs.io

yay! happy it works!