Hi I am trying to access the underlying s3 storage of a repo lakeFS #help

Hi, I am trying to access the underlying s3 stora...

Cristian Caloian

09/12/2023, 9:30 AM

Hi, I am trying to access the underlying s3 storage of a repository using

aws --profile lakefs s3 ls s3://<bucket>/<repo> --recursive

. I have configured the

lakefs

profile with credentials from LakeFS, and the corresponding endpoint url. I also have

AuthFullAccess

FSFullAccess

and

RepoManagementFullAccess

permissions. However, I still get the following error

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Forbidden

Do you have an idea what I am missing?

Idan Novogroder

09/12/2023, 10:07 AM

Hi @Cristian Caloian, Two questions- 1. What do you mean when you say I have configured the
lakefs
profile with credentials from LakeFS? do you mean you use access key and secret from lakeFS or do you mean you use the same AWS credentials your lakeFS instance use? 2. About your command-

aws --profile lakefs s3 ls s3://<bucket>/<repo> --recursive

why do you use s3://<bucket>/<repo>? you should use the storage namespace you configured for your repo, e.g.-

s3://<bucket>/<prefix>

09/12/2023, 10:11 AM

isn't it supposed to be `s3://<repo>/<branch>/<somePath>`` ?

09/12/2023, 10:13 AM

Oh nevermind, by

underlying s3 storage

you mean the back end S3 bucket that LakeFS is using ... in which case you need the credential for that S3 bucket (nothing to do with Lakefs credential)

Idan Novogroder

09/12/2023, 11:22 AM

If you're asking if you should use AWS credentials in your

~/.aws/credentials

file, the answer is yes. Please let me know if it worked for you 🙏🏽

Cristian Caloian

09/12/2023, 12:00 PM

I guess my first question is whether, in the context of migrating a repository from one LakeFS installation to a completely different LakeFS installation, I need to copy the underlying S3 storage. If the answer is yes then I think now I understand that it needs the AWS credentials.

Cristian Caloian

09/12/2023, 12:12 PM

I think I figured out… from the linked notebook in the guide, I think it is the underlying storage namespace

Copy code

source_storage_namespace = '<s3://mySourceBucket>'
target_storage_namespace = '<s3://myTargetBucket>'

aws_region = '<AWS region name>' # e.g. us-east-1
aws_access_key_id = 'aaaaaaaaaaaaa'
aws_secret_access_key = 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb'

Thanks again for making me realize the distinction between the two 🙏

mario luigi dance 1

3 Views

Open in Slack

Previous Next