Hi all i wsa trying to use lakefs free version paired with a lakeFS #help

Hi all i wsa trying to use lakefs free version pai...

Yaphet Kebede

01/10/2024, 3:43 PM

Hi all i wsa trying to use lakefs free version paired with an existing s3 bucket, I had a bucket setup in the following way

Copy code

blockstore:
      type: s3
      s3:
        endpoint: "<https://na-s3.somehost.com/bucket>"
        discover_bucket_region: false
        credentials:
          access_key_id: "######"
          secret_access_key: "########"

the s3 bucket interface i am using is the following (https://docs.netapp.com/us-en/ontap/s3-config/ontap-s3-supported-actions-reference.html#bucket-operations) I was wondering if anyone here is using netapp s3 buckets as storage backend for lakefs? I was getting errors like this when i was using it

Copy code

error="operation error S3: GetObject, https response error StatusCode: 501, RequestID: , HostID: , api error NotImplemented: A header or query you provided implies functionality that is not implemented."

which i believe is coming from my netapp s3 instance, but was wondering what headers were being sent by lakefs ?

Itai Admi

01/10/2024, 4:20 PM

Hey @Yaphet Kebede, I’m not sure if lakeFS supports NetApps, but I can try to help. At which point do you get errors? Is the log provided from lakeFS? Can you provide the full logs of lakeFS during that time? Overall it doesn’t seem like lakeFS is using any advanced headers as part of the request to GetObject.

Yaphet Kebede

01/10/2024, 4:21 PM

it's when i try to create a new repo , specifcally the example repo

👍 1

Yaphet Kebede

01/10/2024, 4:21 PM

let me grab the fill log

Yaphet Kebede

01/10/2024, 4:24 PM

Copy code

time="2024-01-09T22:05:17Z" level=info msg="initialize blockstore adapter" func=pkg/block/factory.BuildBlockAdapter file="build/pkg/block/factory/build.go:32" type=s3
time="2024-01-09T22:05:17Z" level=info msg="initialized blockstore adapter" func=pkg/block/factory.buildS3Adapter file="build/pkg/block/factory/build.go:111" type=s3
time="2024-01-09T22:05:18Z" level=info msg="initialize blockstore adapter" func=pkg/block/factory.BuildBlockAdapter file="build/pkg/block/factory/build.go:32" type=s3
time="2024-01-09T22:05:18Z" level=info msg="initialized blockstore adapter" func=pkg/block/factory.buildS3Adapter file="build/pkg/block/factory/build.go:111" type=s3
time="2024-01-09T22:05:18Z" level=info msg="initialize OpenAPI server" func=pkg/api.Serve file="build/pkg/api/serve.go:38" service=api_gateway
time="2024-01-09T22:05:18Z" level=info msg="initialized S3 Gateway handler" func=pkg/gateway.NewHandler file="build/pkg/gateway/handler.go:124" s3_bare_domain="[<http://s3.local.lakefs.io|s3.local.lakefs.io>]" s3_region=us-east-1
time="2024-01-09T22:05:18Z" level=info msg="starting HTTP server" func=cmd/lakefs/cmd.glob..func8 file="cmd/run.go:307" listen_address="0.0.0.0:8000"

lakeFS 1.1.0 - Up and running (^C to shutdown)...


     ██╗      █████╗ ██╗  ██╗███████╗███████╗███████╗
     ██║     ██╔══██╗██║ ██╔╝██╔════╝██╔════╝██╔════╝
     ██║     ███████║█████╔╝ █████╗  █████╗  ███████╗
     ██║     ██╔══██║██╔═██╗ ██╔══╝  ██╔══╝  ╚════██║
     ███████╗██║  ██║██║  ██╗███████╗██║     ███████║
     ╚══════╝╚═╝  ╚═╝╚═╝  ╚═╝╚══════╝╚═╝     ╚══════╝

│
│ If you're running lakeFS locally for the first time,
│     complete the setup process at <http://127.0.0.1:8000/setup>
│
│
│ For more information on how to use lakeFS,
│     check out the docs at <https://docs.lakefs.io/quickstart/>
│

│
│ For support or any other question,                            >(.＿.)<
│     join our Slack channel <https://docs.lakefs.io/slack>         (  )_
│

Version 1.1.0

time="2024-01-09T22:06:34Z" level=error msg="failed to get S3 object bucket *******/example key dummy" func="pkg/logging.(*logrusEntryWrapper).Errorf" file="build/pkg/logging/logger.go:284" error="operation error S3: GetObject, https response error StatusCode: 501, RequestID: , HostID: , api error NotImplemented: A header or query you provided implies functionality that is not implemented." host=lakefs.local method=POST operation=GetObject operation_id=CreateRepository path=/api/v1/repositories request_id=4dd67131-c2ca-4773-ac99-ba25f0587dfa service_name=rest_api user=admin
time="2024-01-09T22:06:34Z" level=warning msg="Could not access storage namespace" func="pkg/api.(*Controller).CreateRepository" file="build/pkg/api/controller.go:1601" error="operation error S3: GetObject, https response error StatusCode: 501, RequestID: , HostID: , api error NotImplemented: A header or query you provided implies functionality that is not implemented." reason=unknown service=api_gateway storage_namespace="s3://*******/example"
time="2024-01-09T22:06:39Z" level=error msg="failed to get S3 object bucket *******/example key dummy" func="pkg/logging.(*logrusEntryWrapper).Errorf" file="build/pkg/logging/logger.go:284" error="operation error S3: GetObject, https response error StatusCode: 501, RequestID: , HostID: , api error NotImplemented: A header or query you provided implies functionality that is not implemented." host=lakefs.local method=POST operation=GetObject operation_id=CreateRepository path=/api/v1/repositories request_id=dd6419a6-71d8-4176-adaf-ba251ae2e07d service_name=rest_api user=admin
time="2024-01-09T22:06:39Z" level=warning msg="Could not access storage namespace" func="pkg/api.(*Controller).CreateRepository" file="build/pkg/api/controller.go:1601" error="operation error S3: GetObject, https response error StatusCode: 501, RequestID: , HostID: , api error NotImplemented: A header or query you provided implies functionality that is not implemented." reason=unknown service=api_gateway storage_namespace="s3://*******/example"

Oz Katz

01/10/2024, 4:43 PM

Shooting in the dark, but for other S3-compatible storages (namely MinIO) it is typically required to also set

force_path_style

true

to avoid DNS and certificate mismatches.. In your example, the config would look something like this:

Copy code

blockstore:
  type: s3
  s3:
    endpoint: "<https://na-s3.somehost.com/bucket>"
    discover_bucket_region: false
    force_path_style: true  # <-- Add this
    credentials:
      access_key_id: "######"
      secret_access_key: "########"

gratitude thank you 1

Yaphet Kebede

01/10/2024, 4:44 PM

hmm let me try that thanks!

Yaphet Kebede

01/10/2024, 4:57 PM

i am wondering if it the way i am specifying

Storage Namespace

on the UI ? i did

Copy code

Storage Namespace = "<s3://na-s3.somehost.com/bucket/test-repo>"

got the aforementioned error tried

Copy code

Storage Namespace = "<s3://test-repo>"

i got the same error

Copy code

time="2024-01-10T16:48:37Z" level=warning msg="Could not access storage namespace" func="pkg/api.(*Controller).CreateRepository" file="build/pkg/api/controller.go:1601" error="operation error S3: GetObject, https response error StatusCode: 501, RequestID: , HostID: , api error NotImplemented: A header or query you provided implies functionality that is not implemented." reason=unknown service=api_gateway storage_namespace="<s3://test-repo/>"

wondering if the real issue is that i don't understand what this value should be ?

Oz Katz

01/10/2024, 4:58 PM

the value should be in the form of

s3://<bucket name>/<path in the bucket>

Oz Katz

01/10/2024, 4:58 PM

In your second example, it is expected that `

Copy code

test-repo

is a bucket that exists in your S3 (or S3-compatible) storage

Oz Katz

01/10/2024, 5:02 PM

Also, looking at the configuration file you sent,

/bucket

should be removed - the endpoint should not contain a bucket name

Yaphet Kebede

01/10/2024, 5:06 PM

In your second example, it is expected that `

Copy code
test-repo

is a bucket that exists in your S3 (or S3-compatible) storage

No this is the new repo path that i wanted to create, and my s3 has a bucket called

bucket

so you are saying

Copy code

<s3://bucket/test-repo>

should work?

Yaphet Kebede

01/10/2024, 5:07 PM

in lakefs would know to contact the s3 host (na-s3.somehost.com) ?

Itai Admi

01/10/2024, 6:25 PM

Yes to both questions.

gratitude thank you 1

Amit Kesarwani

01/10/2024, 9:21 PM

FYI: I had tested lakeFS with NetApp’s StorageGRID with following configurations and it worked fine out-of-the-box:

Copy code

docker run -d --pull always -p 8000:8000 \
--name lakefs-netapp \
-e LAKEFS_BLOCKSTORE_TYPE='s3' \
-e LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE="true" \
-e LAKEFS_BLOCKSTORE_S3_ENDPOINT="<https://xxx.netapp.com:443>" \
-e LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID="AAAAAAAAA" \
-e LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY="bbbbb" \
treeverse/lakefs run --local-settings

gratitude thank you 1

🙏 1

26 Views

Open in Slack

Previous Next