https://lakefs.io/ logo
Join SlackCommunities
Powered by
Hi all, when I tried to use a third party java lib...
# help
t
Hi all, when I tried to use a third party java library (which we not able to replace with something else), the library does the following call: boolean existsBucket = s3Client.doesBucketExistV2(bucketName); Which is to check if a LakeFS repo exists. However, this code failed with the following exception message: Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: This operation is not supported in LakeFS (Service: Amazon S3; Status Code: 405; Error Code: ERRLakeFSNotSupported; Request ID: 40a1e78b-f23e-4b04-8294-4338345d7c74; S3 Extended Request ID: CB07BFE5E44B0E5F; Proxy: null) Is it something LakeFS that can add support? If yes, that will solve a major problem for us and many thanks for suggestions/walkarounds
Sorry forgot to mention: If we go directly to our MinIO layer, there is no problem. However, since we only want to expose LakeFS as the entrypoint for our data, we really like that LakeFS can resolve this problem
i
Hey @Tongguo Pang, do you know the exact http call that reach the lakeFS server? You can probably get it from lakeFS logs or the ingress service. The reason I'm asking is that it seems like that call performs some operation that is not supported by lakeFS, and HeadBucket which is usually how s3 sdks figure out if a bucket exists, is supported
t
Hi @Itai Admi, thanks for getting back to me. I am using the LakeFS docker image, I am not able to see the HTTP request from the docker logs. And I am not able to find where the log might be after I SSH into my docker container. Maybe you can point me to the exact logging location?
Also, I guess the same thing that there is something LakeFS doesn't support, still, would it be possible that LakeFS can support? We really don't want our develoeprs to go to MnIO directly, even though that seems to be only workaround now
If LakeFS can add support for this operation, we would like to disallow accessing to MinIO again
i
We don’t know yet, it’s possible that SDK needs some more configuration options. Is there any operation that you’re able to perform with this SDK and lakeFS?
t
We are not able to go further as this is the entrypoint for the SDK. However, we don't have any problems with LakeFS and our Spark and Trino applications
so this might be a very small fix if any
i
Cool - can you share how you’ve configured the SDK to work with lakeFS?
Which SDK is it btw?
t
DolphinScheduler
We need to give DolphinScheduler the access key, secret key, endpoint of lakefs, as well as a root repo/bucket for storage purpose
when DS starts up, it will use the above credentails to check if the repo/bucket exists. And this is where we failed and DS will not start
BTW, I don't see the /var/log directory in the container, and also there isn't a log dir in the /home/lakefs directory, so I am not able to guess where the logging goes
i
I believe this is the failure you’re getting: https://github.com/aws/aws-sdk-java/issues/2889
Would you mind opening a new feature request here: https://github.com/treeverse/lakeFS/issues/new/choose?
t
I also found this on internet.
OK I will add this feature request. it will be greatly helpful if this is addressed on your side
gratitude thank you 1
👍 1
arreciate your quick response
lakefs 1
Just added the FR, hope your team can address this quickly 🙂
thanks again for your help
lakefs 1
a
Thanks for the FR! Let's continue on the issue.
t
Thanks @Ariel Shaqed (Scolnicov) for taking up this issue. Very much appreciated for LakeFS team to consider this request. It will be very helpful if you can add this support
sunglasses lakefs 1
as I mentioned above, I really don't like to expose our MinIO layer to client calls
3 Views
Open in Slack
PreviousNext
Powered by