Hi! I was looking to use the `gateways.s3.fallback...
# help
Hi! I was looking to use the
setting, but haven't been able to figure out exactly how to use it. Is there any documentation besides the configuration reference guide? In particular, I'm unsure how to pass authentication credentials to the fallback backend: If I specify credentials that don't exist in lakeFS but are valid on the fallback S3 endpoint, I get an
error returned by the
aws s3
command. If I specify the lakeFS credentials and try to access a bucket that would be handled by the fallback case, I get an
error. Do the credentials need to be identical for both the lakeFS S3 endpoint and the fallback? Thanks for your help! ๐Ÿ™
๐Ÿ‘€ 1
Hey @Adrian Rumpold! Looking at the code, youโ€™re absolutely right. We have a bug where the credentials are being checked by lakeFS before proxied to to the fallback. Since these creds are of S3, lakeFS will indeed fail to authorize that request. Can you open a bug please?
As a short term fix, creating a user with the same access and secret keys in lakeFS should work. Hereโ€™s a plumbing (unsafe for prod usage) command you can use with lakefs binary:
Copy code
lakefs superuser
Error: required flag(s) "user-name" not set
  lakefs superuser [flags]

      --access-key-id string       create this access key ID for the user (for ease of integration)
  -h, --help                       help for superuser
      --secret-access-key string   use this access key secret (potentially insecure, use carefully for ease of integration)
      --user-name string           an identifier for the user (e.g. "jane.doe")
Thanks for getting back to me so quickly! jumping lakefs I'll open an issue, meanwhile I can work around by having identical credentials for both lakeFS and the fallback (working on a prototype only)
I've created an issue in Github: https://github.com/treeverse/lakeFS/issues/5359
๐Ÿ™Œ 1
๐Ÿ™Œ๐Ÿฝ 1
Hey @Adrian Rumpold Thanks for reporting and opening the issue! I would really appreciate it if you could elaborate a bit about your use case for this feature? what problem does it solve for you?
Sure can! We are trying to use lakeFS as an S3 backend for Flyte workflows (https://flyte.io), looking for a way to transparently version their
resources. However, Flyte also puts its own metadata into the same S3 endpoint, which we obviously don't want to end up in lakeFS. So we could either add an S3 request router in front of lakeFS, or use the fallback feature in lakeFS.
Understood - thank you so much! very helpful. Would you be interested in contributing to a how-to guide for the Flyte/lakeFS integration (similar to the Kubeflow guide)? ๐Ÿค—
We're not quite there yet, but I'll keep it in mind!
๐Ÿ‘ 2