https://lakefs.io/ logo
Join SlackCommunities
Powered by
Hi! Hopefully I am not bothering. So ... trying t...
# help
o
Hi! Hopefully I am not bothering. So ... trying to play with LakeFS and I am probably doing something unreasonably unintelligent. I was able to configure without much issue a local repository. But I am not having luck with configuring lakefs with minio. Minio seems to be configured correctly. I can use the `mc`tool. Which also should mean the credentials are fine. But I am getting a signature does not match error on the LakeFS side.
Untitled
Untitled
Any ideas on how do I debug this ?
a
Hi Ori, Thanks for the clear description. I'll get on it in the morning. In the meanwhile, please verify that the credentials in the lakeFS config file are for S3.
o
there’s also a minio example in the docs, if it helps: https://docs.lakefs.io/reference/configuration.html#example-minio
a
Hi Ori, So my message from last night seems garbled when I reread it. Sorry about that. Please check your lakeFS configuration - the config file that lakefs reads on startup. It should have appropriate aws-style credentials for your minio system.
blockstore.s3.credentials, see e.g. the sample configuration sent by @Oz Katz above. Please let me know how you get along, here on channel (you're probably not going to be the only one with this issue) or by dm if more convenient for you. Thanks!
o
here is my config file:
Copy code
cat .lakefs.yaml
---
listen_address: "0.0.0.0:8888"
database:
  connection_string: "<postgres://main:main@postgresdatabase.internal:5432/main?sslmode=disable>"
#blockstore:
#  type: "local"
#  local:
#    path: "~/lakefs_data"
auth:
  encrypt:
    # we need to figure out a way to get this from PLATFORM_PROJECT_ENTROPY
    secret_key: "CHANGEME-ZJVRI5JFZN6XXUGWEKADTCYBTPDPORH7ADEWTSO2ZKGGCYS2HG5A"
blockstore:
  type: s3
  s3:
    region: us-east-1
    force_path_style: true
    endpoint: <https://s3.master-7rqtwti-u2fgs74krpwaa.eu-3.platformsh.site>
    credentials:
      access_key_id: admin
      access_secret_key: minioadmin
(don’t worry about credentials in public slack .. this is a throway example I will destroy with prejudice shortly).
👀 1
I can confirm the credentials are ok with
Copy code
mc alias set miniolake <https://s3.master-7rqtwti-u2fgs74krpwaa.eu-3.platformsh.site> admin minioadmin
and
Copy code
mc admin trace miniolake
Maybe the problem is with the `
Copy code
gateways:
`
I was not sure if that part was required or not ?
a
Yes, it's needed. Not sure it affects your flow, but could you please add it?
I am unable to reproduce your issue when running against a minio in a local Docker container that runs the standard container 
minio/minio
(from DockerHub), with or without the 
gateways
stanza in lakeFS config. I tried accessing it with S3 v2 and v4 signatures, no issues there either. Is there something special about your minio configuration, perhaps?
o
My whole minio thing is basically:
Copy code
wget -N <https://dl.min.io/server/minio/release/linux-amd64/minio>
          chmod +x minio
          mv minio ~/bin/
Copy code
start: ~/bin/minio server --address ":$PORT"  ~/data/files
I am going to try with the 
gateways
property
👍 1
Oh maybe I should run the minio with the gatway option ?
Sounds weird.
a
I think 
gateways
is a red herring right now, and prefer to ignore it (for now).
May I access your minio instance to see whether I can reproduce against it?
o
yes of course ..
Copy code
<http://s3.master-7rqtwti-u2fgs74krpwaa.eu-3.platformsh.site|s3.master-7rqtwti-u2fgs74krpwaa.eu-3.platformsh.site> Authorization: AWS4-HMAC-SHA256 Credential=admin/20201201/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=4d619538ded157bf48241cad8f7192fc335b0ce06cb59ba6c62364361a1d9187
<http://s3.master-7rqtwti-u2fgs74krpwaa.eu-3.platformsh.site|s3.master-7rqtwti-u2fgs74krpwaa.eu-3.platformsh.site> Authorization: AWS4-HMAC-SHA256 Credential=admin/20201201/us-east-1/s3/aws4_request, SignedHeaders=content-encoding;expect;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length, Signature=0a4957240fb29d7f1c531a127ddade3033feb3fe1d3c278f8265ca70758f18f6```
The first is a successful request (from the mc client) .. the second from lakefs fails ... the difference seems to be in the fields that are signed ...
The other difference seems to be the content encoding .. where the lakefs is using 
Content-Encoding: aws-chunked
.. and signing the payload while the mc tool seems to be 
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
vs 
X-Amz-Content-Sha256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
Is it useful to keep the test cluster configured for minio ? (I can keep the minio running - I just want to configure it for local). And a subsdiary lazy question : can you have multiple blockstores active at the same time ? It is a bit weird that there is no error in validating the yaml when there are duplicate keys ...
If I remember correctly you had some ideas about where this might have come from .. ? And an idea for a solution ?
Untitled
Hi, I cam back to testing this .. it still seems to behave the same way with a 403 and 
SignatureDoesNotMatch
... I am now on v0.30.0
3 Views
Open in Slack
PreviousNext
Powered by