Hey guys, got an error which I am not sure how to bypass. From your docs, I am running the following command and pointing to an s3 location that we have:

podman run --pull always -p 8000:8000 \

-e LAKEFS_BLOCKSTORE_TYPE='s3' \

-e LAKEFS_BLOCKSTORE_S3_ENDPOINT='<https://s3.echo.stfc.ac.uk>'    \

-e AWS_ACCESS_KEY_ID='####' \

-e AWS_SECRET_ACCESS_KEY='#####' \

treeverse/lakefs run --local-settings

All of this is correct, but when we go to create the LakeFS repo, pointing to

<s3://mast/>

, on the UI it states it cannot find the storage namespace. Looking in the logs we can see it's because it is trying to get to:

<https://mast.s3.echo.stfc.ac.uk>

When in reality, the storage is:

<https://s3.echo.stfc.ac.uk/mast>

Is there a setting we are missing here? Bit stumped as to why the storage namespace is chucked infront of the s3 endpoint!

Hey James, that’s the bucket virtual address vs. path style. Add

-e LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE=true

to your setup

More information about this here. This has been a pain point with S3 access for many years now.

Hi both, thank you. I did try the above before coming to Slack and it still would not connect properly to our bucket. This is what the logs spits out:

time="2024-08-20T20:27:49Z" level=error msg="failed to get S3 object bucket mast key dummy" func="pkg/block/s3.(*Adapter).Get" file="build/pkg/block/s3/adapter.go:357" error="operation error S3: GetObject, https response error StatusCode: 403, RequestID: tx000000000000004867939-0066c5d4b5-2f4b27365-default, HostID: 2f4b27365-default-default, api error RequestTimeTooSkewed: UnknownError" host="127.0.0.1:8000" method=POST operation=GetObject operation_id=CreateRepository path=/api/v1/repositories user=admin
time="2024-08-20T20:27:49Z" level=warning msg="Could not access storage namespace" func="pkg/api.(*Controller).CreateRepository" file="build/pkg/api/controller.go:2004" error="operation error S3: GetObject, https response error StatusCode: 403, RequestID: tx000000000000004867939-0066c5d4b5-2f4b27365-default, HostID: 2f4b27365-default-default, api error RequestTimeTooSkewed: UnknownError" reason=unknown service=api_gateway storage_namespace="<s3://mast/>"

Not sure what to decipher from that!

To clarify - is the log above from a run with

-e LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE=true

set?

yes, sorry shout have clarified that, full input:

podman run --pull always -p 8000:8000 \
   -e LAKEFS_BLOCKSTORE_TYPE='s3' \
   -e LAKEFS_BLOCKSTORE_S3_ENDPOINT='<https://s3.echo.stfc.ac.uk>'        \
   -e AWS_ACCESS_KEY_ID='#####' \
   -e LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE=true \
   -e AWS_SECRET_ACCESS_KEY='####' \
   treeverse/lakefs run --local-settings

Strange... It's complaining about a time difference. Perhaps the time inside your lakeFS container is not synced, and is too far from the time on your S3 storage? (Or vice versa ...?) I do remember when some container runtimes would skew times like crazy; I was hoping those days are gone.

I am not sure if it will solve the problem but try changing

AWS_ACCESS_KEY_ID

and

AWS_SECRET_ACCESS_KEY

environment variables to

LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID

and

LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY

respectively.

good morning both, thank you for your replies. @Amit Kesarwani I tried your suggestion but I'm getting the same error as before. @Ariel Shaqed (Scolnicov) I'll try look into this time difference and get back to you both! If you have any other suggestions let me know

That's good news. Thanks for updating

If you're losing time sync on a container then it will happen again! Consider adding an ntpd into the container. I seem to remember this being a particular pain on MacOS Docker Desktop - here's an old article of that helps. Not necessarily limited to MacOS/Docker.

Will do! Yes I am on MacOS and found the same as you, seems to be a common problem