Channels
announcements
blockers_for_windward
career-opportunities
celebrations
cuddle-corner
data-discussion
data-events
dev
events
general
help
iceberg-integration
lakefs-for-beginners
lakefs-hubspot-cloud-registration-email-automation
lakefs-releases
lakefs-suggestions
lakefs-twitter
linen-dev
memes-and-banter
new-channel
new-channel
say-hello
stackoverflow
test
Title
r
Raphaël
12/20/2022, 4:20 PMHi everyone,
I've running lakefs locally with docker to try to make it communicate with a minio s3 bucket. No problem for the installation and the creation of the bucket but when i want to create a new repo on lakefs to retrieve data i get the error : ' failed to create repository: failed to access storage '.
Do you have any ideas that may be the cause of this problem ?
Thanks in advance.
j
Jonathan Rosenberg
12/20/2022, 4:27 PMHi Raphael!
Can you share your lakeFS configurations?
it seems that lakeFS cannot reach your Minio server. Can you make sure that you provided the right credentials and endpoint as specified here?
r
Raphaël
12/20/2022, 4:38 PMThanks for your answer.
How can i access this file for provide the right credentials ?
j
Jonathan Rosenberg
12/20/2022, 5:19 PMSo the default behavior of running the lakeFS docker image will work with the local filesystem as the storage backend.
Run it with the following configuration:
docker run --pull always -p 8000:8000 \
-e LAKEFS_BLOCKSTORE_TYPE='s3' \
-e LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE='true' \
-e LAKEFS_BLOCKSTORE_S3_ENDPOINT='http://<minio_endpoint>' \
-e LAKEFS_BLOCKSTORE_S3_DISCOVER_BUCKET_REGION='false' \
-e LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID='<minio_access_key>' \
-e LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY='<minio_secret_key>' \
treeverse/lakefs run --local-settingsd
Deepak Rangarao
12/20/2022, 5:32 PM@Jonathan Rosenberg if minio is also running locally as a container, should we use http://host.docker.internal:9001 as the endpoint ?
j
Jonathan Rosenberg
12/20/2022, 5:34 PMIf they run on the same network then yes
d
Deepak Rangarao
12/20/2022, 5:37 PMI get this error @Jonathan Rosenberg
time="2022-12-20T17:37:10Z" level=warning msg="Could not access storage namespace" func="pkg/api.(*Controller).CreateRepository" file="build/pkg/api/controller.go:1380" error="InvalidArgument: S3 API Requests must be made to API port.\n\tstatus code: 400, request id: , host id: " reason=unknown service=api_gateway storage_namespace="<s3://host.docker.internal:9001/lakefs>"I see a few warnings on startup
time="2022-12-20T17:35:30Z" level=warning msg="Tried to to get AWS account ID for BI" func="pkg/cloud/aws.(*MetadataProvider).GetMetadata.func1" file="build/pkg/cloud/aws/metadata.go:58" error="SerializationError: failed to unmarshal error message\n\tstatus code: 400, request id: \ncaused by: UnmarshalError: failed to unmarshal error message\n\t00000000 3c 3f 78 6d 6c 20 76 65 72 73 69 6f 6e 3d 22 31 |<?xml version=\"1|\n00000010 2e 30 22 20 65 6e 63 6f 64 69 6e 67 3d 22 55 54 |.0\" encoding=\"UT|\n00000020 46 2d 38 22 3f 3e 3c 45 72 72 6f 72 3e 3c 43 6f |F-8\"?><Error><Co|\n00000030 64 65 3e 49 6e 76 61 6c 69 64 41 72 67 75 6d 65 |de>InvalidArgume|\n00000040 6e 74 3c 2f 43 6f 64 65 3e 3c 4d 65 73 73 61 67 |nt</Code><Messag|\n00000050 65 3e 53 33 20 41 50 49 20 52 65 71 75 65 73 74 |e>S3 API Request|\n00000060 73 20 6d 75 73 74 20 62 65 20 6d 61 64 65 20 74 |s must be made t|\n00000070 6f 20 41 50 49 20 70 6f 72 74 2e 3c 2f 4d 65 73 |o API port.</Mes|\n00000080 73 61 67 65 3e 3c 52 65 71 75 65 73 74 49 64 3e |sage><RequestId>|\n00000090 30 3c 2f 52 65 71 75 65 73 74 49 64 3e 3c 2f 45 |0</RequestId></E|\n000000a0 72 72 6f 72 3e |rror>|\n\ncaused by: unknown error response tag, {{ Error} []}"not sure if they matter
j
Jonathan Rosenberg
12/20/2022, 5:46 PMPlease try using port 9000
👍🏼 1
a
Ariel Shaqed (Scolnicov)
12/20/2022, 5:46 PMHi Deepak,
Let me try to help please. Your storage namespace might be a bit odd. AFAIR MinIO by default uses port 9000 for the S3 protocol -- and the error message indicates that it might be 9001 in your configuration?
d
Deepak Rangarao
12/20/2022, 5:50 PMstill getting an error with 9000, let me DM you the exact docker run so you can see what I am missing … I am sure I am doing something silly
Yes, it worked ! mostly user error but here is what I did
• @Jonathan Rosenberg @Ariel Shaqed (Scolnicov) you both were right, I should have used port 9000
• The 2nd mistake I did (unique situation) was my 2 containers were in different networks
Once I fixed the above 2 issues it worked fine
👍 2
👍🏻 1
:jumping-lakefs: 2
a
Ariel Shaqed (Scolnicov)
12/20/2022, 6:03 PMCool! Hope you have fun in our lake, and let us know how you get along. 😒unglasses_lakefs:
👍 1
r
Raphaël
12/21/2022, 8:50 AMHi @Jonathan Rosenberg,
Let me summarize the different steps we were talking about yesterday :
1. Creation of the bucket on minio with the change of bucket policy
2. Launch lakefs with the credentials file with minio accesses
3. Creation of the lakefs repo
It's right ?
j
Jonathan Rosenberg
12/21/2022, 8:52 AMHi @Raphaël
Yes, although it’s not a credentials file, but credential environment variables…
r
Raphaël
12/21/2022, 9:30 AMOf course, excuse my lack of precision.
Even following theses steps, i still get the error message : 'failed to create repository: failed to access storage ' ...
j
Jonathan Rosenberg
12/21/2022, 9:31 AMCan you share the env vars with me?
You can DM them if you want…
👍 1