Thread (2): how does the GUI access data? When you log into the lakeFS GUI, you get a jwt representing your identity. The GUI uses that for all access, and the lakeFS server will apply access policies for that user. Now the GUI can access objects on lakeFS. It uses the lakeFS api for that. Every metadata operation (such as listing repos or branches, or listing paths in a "directory") is obviously performed on the server. What about data operations (uploading and downloading contents)? The GUI still asks the lakeFS server. The server applies policies for the user, and if all goes well it accesses data on the storage backend (S3) on behalf of the user and transfers data to or from the user. This is all done using the server credentials; the user does not need credentials for direct data access on the storage.

Ok so essentially it sounds like I need to create an IAM_ROLE for my server hosting lakeFS which will directly access the s3 buckets…when a user logs in they get an abstraction layer through lakeFS e.g “lakefs://“ set of permissions that can access the data

Yup! Typically your server will run on aws ec2, or on k8s, both of which allow you to give it instance roles. Have you taken a look at our setup trail? It starts with "preparing your storage" (link goes to s3), which helpfully includes an iam policy that will work.

I saw that, but that looks like a bucket policy, not a iam policy?

Hi Kevin, This is odd. It looks like you comfigured everything according to this guide for EC2. Maybe you have some environment variable setting a profile? If you need more help, could you send the config YAML file and environment variables starting with "AWS"? But please... Make sure not to send any secrets, of course!

ohhhh i see it…yeah ‘profile: default’ was in my config

I saw that, but that looks like a bucket policy, not a iam policy?

Sorry, missed this one. This bucket policy will work. You're right, I don't think we have a guide for setting a role policy anywhere. You'll need to copy the same permissions, of course. I'll look on Sunday to see if I can find an example written out somewhere.

it's awful and confusing as hell

You'd wish it did that. I think that the default credentials provider goes to the instance metadata service, which appears by virtual magic, and that gives credentials to get the role. Fun fun fun.

aka “magic”

I like prefixes :-) . So I'd recommend something with "lakeFS/repos" in it. That way it's easier for me to know what I and my users put into S3, and what lakeFS put in there.

got it