Hi I was looking to use the `gateways s3 fallback url` setti

Question

Hi I was looking to use the `gateways s3 fallback url` setting but haven t been able to figure out exactly how to use it Is there any documentation besides the <https docs lakefs io reference configuration html|configuration reference guide> In particular I m unsure how to pass authentication credentials to the fallback backend If I specify credentials that don t exist in lakeFS but are valid on the fallback S3 endpoint I get an `InvalidAccessKeyId` error returned by the `aws s3` command If I specify the lakeFS credentials and try to access a bucket that would be handled by the fallback case I get an `AccessDenied` error Do the credentials need to be identical for both the lakeFS S3 endpoint and the fallback Thanks for your help pray

Itai Admi · Answer

Hey < Adrian Rumpold> Looking at the code you re absolutely right We have a bug where the credentials are being checked by lakeFS before proxied to to the fallback Since these creds are of S3 lakeFS will indeed fail to authorize that request Can you open a bug please

Itai Admi · Answer

As a short term fix creating a user with the same access and secret keys in lakeFS should work Here s a plumbing unsafe for prod usage command you can use with lakefs binary ```lakefs superuser Error required flag s user name not set Usage lakefs superuser flags Flags access key id string create this access key ID for the user for ease of integration h help help for superuser secret access key string use this access key secret potentially insecure use carefully for ease of integration user name string an identifier for the user e g jane doe ```

Adrian Rumpold · Answer

Thanks for getting back to me so quickly jumping lakefs I ll open an issue meanwhile I can work around by having identical credentials for both lakeFS and the fallback working on a prototype only

Adrian Rumpold · Answer

I ve created an issue in Github <https github com treeverse lakeFS issues 5359>

Oz Katz · Answer

Hey < Adrian Rumpold> Thanks for reporting and opening the issue I would really appreciate it if you could elaborate a bit about your use case for this feature what problem does it solve for you

Adrian Rumpold · Answer

Sure can We are trying to use lakeFS as an S3 backend for Flyte workflows <https flyte io> looking for a way to transparently version their `FlyteFile` resources However Flyte also puts its own metadata into the same S3 endpoint which we obviously don t want to end up in lakeFS So we could either add an S3 request router in front of lakeFS or use the fallback feature in lakeFS

Oz Katz · Answer

Understood thank you so much very helpful Would you be interested in contributing to a how to guide for the Flyte lakeFS integration similar to the <https docs lakefs io integrations kubeflow html|Kubeflow guide> hugging face

Adrian Rumpold · Answer

We re not quite there yet but I ll keep it in mind