https://lakefs.io/ logo
#help
Title
# help
a

Adrian Rumpold

02/28/2023, 9:01 AM
Hi! I was looking to use the
gateways.s3.fallback_url
setting, but haven't been able to figure out exactly how to use it. Is there any documentation besides the configuration reference guide? In particular, I'm unsure how to pass authentication credentials to the fallback backend: If I specify credentials that don't exist in lakeFS but are valid on the fallback S3 endpoint, I get an
InvalidAccessKeyId
error returned by the
aws s3
command. If I specify the lakeFS credentials and try to access a bucket that would be handled by the fallback case, I get an
AccessDenied
error. Do the credentials need to be identical for both the lakeFS S3 endpoint and the fallback? Thanks for your help! ๐Ÿ™
๐Ÿ‘€ 1
i

Itai Admi

02/28/2023, 9:37 AM
Hey @Adrian Rumpold! Looking at the code, youโ€™re absolutely right. We have a bug where the credentials are being checked by lakeFS before proxied to to the fallback. Since these creds are of S3, lakeFS will indeed fail to authorize that request. Can you open a bug please?
As a short term fix, creating a user with the same access and secret keys in lakeFS should work. Hereโ€™s a plumbing (unsafe for prod usage) command you can use with lakefs binary:
Copy code
lakefs superuser
Error: required flag(s) "user-name" not set
Usage:
  lakefs superuser [flags]

Flags:
      --access-key-id string       create this access key ID for the user (for ease of integration)
  -h, --help                       help for superuser
      --secret-access-key string   use this access key secret (potentially insecure, use carefully for ease of integration)
      --user-name string           an identifier for the user (e.g. "jane.doe")
a

Adrian Rumpold

02/28/2023, 9:55 AM
Thanks for getting back to me so quickly! jumping lakefs I'll open an issue, meanwhile I can work around by having identical credentials for both lakeFS and the fallback (working on a prototype only)
I've created an issue in Github: https://github.com/treeverse/lakeFS/issues/5359
๐Ÿ™Œ 1
๐Ÿ™Œ๐Ÿฝ 1
o

Oz Katz

02/28/2023, 11:06 AM
Hey @Adrian Rumpold Thanks for reporting and opening the issue! I would really appreciate it if you could elaborate a bit about your use case for this feature? what problem does it solve for you?
a

Adrian Rumpold

02/28/2023, 11:13 AM
Sure can! We are trying to use lakeFS as an S3 backend for Flyte workflows (https://flyte.io), looking for a way to transparently version their
FlyteFile
resources. However, Flyte also puts its own metadata into the same S3 endpoint, which we obviously don't want to end up in lakeFS. So we could either add an S3 request router in front of lakeFS, or use the fallback feature in lakeFS.
o

Oz Katz

02/28/2023, 11:36 AM
Understood - thank you so much! very helpful. Would you be interested in contributing to a how-to guide for the Flyte/lakeFS integration (similar to the Kubeflow guide)? ๐Ÿค—
a

Adrian Rumpold

02/28/2023, 12:44 PM
We're not quite there yet, but I'll keep it in mind!
๐Ÿ‘ 2
13 Views