https://lakefs.io/ logo
Join Slack
Powered by
Hi Team, I have setup lakefs with block adapter lo...
# help
r
Hi Team, I have setup lakefs with block adapter local. Now, I am trying to create repository in OCI bucket (oracle cloud) which is s3 compatible. Did anyone try this before? is there any document to configure repository with OCI bucket?
b
hi, @Raghavendra Hegde - didn't configured OCI bucket before but I can help you.
lakefs works with s3 client sdk so if the compatibility works, OCI should work. I would go with how lakefs connects to MinIO as it is also S3 compatible.
r
ok, thank you @Barak Amar. I will go through link and get back to you
b
You will need to setup lakeFS from the start - but this time use the blockstore type will be s3 as in the document above
Or delete the current respository
r
ok.
I am getting below error. it is something related to SNI support.
ERROR [2021-10-06T124113Z]lakeFS/pkg/block/s3/client_cache.go:69 pkg/block/s3.(*ClientCache).getBucketRegion failed to get region for bucket, falling back to default region error="RequestError: send request failed\ncaused by: x509: certificate is valid for swiftobjectstorage.me-jeddah-1.oraclecloud.com, not lakefs-demo.mynamespace-oci.compat.objectstorage.me-jeddah-1.oraclecloud.com"
Copy code
blockstore:
  type: s3
  s3:
    region: me-jeddah-1
    force_path_style: true
    endpoint: <http://lakefs-demo.mynamespace-oci.compat.objectstorage.me-jeddah-1.oraclecloud.com|lakefs-demo.mynamespace-oci.compat.objectstorage.me-jeddah-1.oraclecloud.com>
    credentials:
      access_key_id: "mykey"
      secret_access_key: "myskey"
this is the configuration that I used
b
the endpoint is usually http or https address and optional port
based on the error it looks like you need to add https
r
ok, thanks. let me check
There is no option or parameter to mention the OCI bucket name. I tried removing bucket name from the endpoint i.e. https://mynamespace-oci.compat.objectstorage.me-jeddah-1.oraclecloud.com. I got below error ERROR [2021-10-06T134449Z]lakeFS/pkg/block/s3/adapter.go:233 pkg/block/s3.(*Adapter).streamToS3 bad S3 PutObject response error="s3 error: <?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Message>STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported
b
The bucket name will be mapped to the repository storage namespace
This is the error you are getting when you try to create repository?
r
yes, I am getting this while creating the repository as well starting the lakefs service
while starting the service lakefs is checking the region.
ERROR [2021-10-06T134940Z]lakeFS/pkg/block/s3/client_cache.go:69 pkg/block/s3.(*ClientCache).getBucketRegion failed to get region for bucket, falling back to default region
👀 1
b
can you set the lakefs s3 region to 'us-east-1'
r
ok
WARNING[2021-10-06T141826Z]lakeFS/pkg/cloud/aws/metadata.go:47 pkg/cloud/aws.(*MetadataProvider).GetMetadata %v: failed to get AWS account ID for BI    error="SerializationError: failed to unmarshal error message\n\tstatus code: 400, request id: \ncaused by: UnmarshalError: failed to unmarshal error message
👀 1
b
no other error
repository was created?
r
sorry, this is the error. repository not created
Copy code
ERROR  [2021-10-06T14:18:42Z]lakeFS/pkg/block/s3/adapter.go:233 pkg/block/s3.(*Adapter).streamToS3 bad S3 PutObject response  
  error="s3 error: <?xml version=\"1.0\" encoding=\"UTF-8\"?><Error><Message>STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported.
  </Message><Code>InvalidRequest</Code></Error>" host="193.122.66.125:8001" 
  method=POST operation=PutObject path=/api/v1/repositories request_id=b2069245-6859-4e0a-a060-82d73e43c516 
  service_name=rest_api status_code=400 url="<https://mynamespace-oci.compat.objectstorage.me-jeddah-1.oraclecloud.com/lakefs-demo/dummy>"
👀 1
b
@Raghavendra Hegde I'm trying to setup similar environment and get to the same point you got - hope to update in couple of hours my progress
r
@Barak Amar sure, np. thanks for your kind gesture👍
b
@Raghavendra Hegde it seems that Oracle Object Storage doesn't support the current method we use to upload an object. The error return from the API 
STREAMING-AWS4-HMAC-SHA256-PAYLOAD is not supported
. When we create a repository we try to upload and download a dummy file to understand if we can work with the underlying storage. In this case the Put request failed. lakeFS uses this method in order to stream the object content while you upload a file to lakeFS - this is enabled with chunked requests. Later I've found https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm where under 'Here is an example of configuring AWS SDK for Java to use Object Storage' I've found that they ask to set 
disableChunkedEncoding
(we are not using java) but this is the method we currently use. I'll open an issue to capture the above, and update tomorrow about time frame we can address it.
Feel free to add more information to the issue.
r
Sure. Thanks a lot for your support. Yes, time frame will help us to make some decisions.
3 Views
Open in Slack
PreviousNext
Powered by