https://lakefs.io/ logo
Join Slack
Channels
Powered by Linen
Title
r

Ronnie Ning

11/16/2022, 6:28 PM
Anyone knows how to set config.yaml for aws dynamodb? I used the example on lakefs for deploy on aws, I got 
ValidationException: The provided key element does not match the schema
i

Itai Admi

11/16/2022, 6:33 PM
Hey Ronnie, I'll try to help with this. Are you getting this error from lakeFS? Can you share the logs (without truncation)?
r

Ronnie Ning

11/16/2022, 6:44 PM
Sorry, just solved this issue. Solution is that we don't need to create a dynamodb table first before we start lakefs. Lakefs will automatically create one for you as long as it has the right role. Thanks Itai
:jumping-lakefs: 1
i

Itai Admi

11/16/2022, 6:46 PM
🙏 Let me know if you need anything else
r

Ronnie Ning

11/16/2022, 8:45 PM
BTW, the folder in S3 bucket generated by LakeFS is dummy by default. Is there a way to customize it?
i

Itai Admi

11/16/2022, 8:49 PM
What do you mean by “dummy by default”?
r

Ronnie Ning

11/16/2022, 8:55 PM
Folder name in the bucket is called 
dummy
i

Itai Admi

11/16/2022, 9:07 PM
dummy
is an object lakeFS creates inside the storage namespace to validate it has access.
r

Ronnie Ning

11/17/2022, 3:13 PM
Do you know what kind of permissions I have to have in order to make lakefs work? I set s3 full access, kms, and dynamodb full access, but I still fail to upload files in lakefs.
i

Itai Admi

11/17/2022, 3:16 PM
What is the error you're getting? Did you successfully create a repo? I find this doc very useful for storage permissions
r

Ronnie Ning

11/17/2022, 3:17 PM
I can create repo, branch, but fail to upload files
i

Itai Admi

11/17/2022, 3:18 PM
That's wierd, can you share the logs?
r

Ronnie Ning

11/17/2022, 3:20 PM
how do we use kms when uploading a file?
in gui, there is no way to use kms, right? We can only use code, for example, lakefs-python package to upload files after kms, right?
i

Itai Admi

11/17/2022, 3:22 PM
What's kms? 😅
r

Ronnie Ning

11/17/2022, 3:23 PM
AWS Key Management Service
to encrypt or decrypt data
i

Itai Admi

11/17/2022, 3:25 PM
In s3 you can configure bucket level encryption, every object stored in the bucket will seamlessly be stored encrypted. You don't need to do it yourself before storing the data in lakeFS.
r

Ronnie Ning

11/17/2022, 3:28 PM
yeah, already enabled default encryption
in our s3 bucket policy, there is one manditory rule, data has to be encrypted using kms before uploading. Maybe that's why my uploading was failed. So I can not use gui to upload data, but have to do it in code.
i

Itai Admi

11/17/2022, 3:32 PM
The UI enables you to upload data from your local storage, so unless you have the encrypted files stored locally, I guess you're right.
r

Ronnie Ning

11/17/2022, 3:33 PM
Thanks
i

Itai Admi

11/17/2022, 3:34 PM
🙏
r

Ronnie Ning

11/21/2022, 9:45 PM
There is no way to have some kind of kms setup in lakefs, right?
y

Yoni Augarten

11/21/2022, 9:53 PM
Hey @Ronnie Ning, I'm not sure what you're asking. If you're asking whether lakeFS can encrypt your objects on S3 using KMS with server-side encryption, then it's currently not supported.
r

Ronnie Ning

11/21/2022, 10:03 PM
I tried several ways to encrypt file, then uploaded it to s3 which is enabled sse, all were failed. But if I put kms as an extra config, it was successful. So I wonder if lakefs can also set kms as config when loading files to s3.
y

Yoni Augarten

11/21/2022, 10:10 PM
Unfortunately, this is not supported at the moment. You're welcome to open an issue describing the use case.
r

Ronnie Ning

11/21/2022, 10:13 PM
Iddo Avneri 10/26/2022, 12:18 PM You are welcome to run lakeFS on top of a bucket with default SSE - it will work and all the data will be encrypted.
y

Yoni Augarten

11/21/2022, 10:14 PM
I wasn't aware of the option to turn on SSE by default. Let me check.
r

Ronnie Ning

11/21/2022, 10:14 PM
I wonder how since I followed the instructions :1. Prepare your encrypted storage. 2. Run lakeFS locally against your storage, but no lucky
y

Yoni Augarten

11/21/2022, 10:15 PM
https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucket-encryption.html
Following these instructions will cause all new objects on your bucket to be encrypted by default. I hope this helps.
r

Ronnie Ning

11/21/2022, 10:17 PM
Yeah, I did enable the default sse on s3, but lakefs file loading failed with access denied.
y

Yoni Augarten

11/21/2022, 10:18 PM
Please provide the complete error that you are seeing
r

Ronnie Ning

11/21/2022, 10:24 PM
HTTP response headers: HTTPHeaderDict({'Date': 'Mon, 21 Nov 2022 22:24:05 GMT', 'Content-Type': 'application/json', 'Content-Length': '383', 'Connection': 'keep-alive', 'X-Content-Type-Options': 'nosniff', 'X-Request-Id': 'aa3c2df7-e081-49fd-9c34-115cf8658ddf'})
HTTP response body: {"message":"s3 error: \u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cError\u003e\u003cCode\u003eAccessDenied\u003c/Code\u003e\u003cMessage\u003eAccess Denied\u003c/Message\u003e\u003cRequestId\u003eAE8C2RC4F6A8A6HV\u003c/RequestId\u003e\u003cHostId\u003eZ1CXhP9KOwEUgaijq3Y2TFZDXihy2G9jQDcDeIMZXAm45WrSc859m7jAyZYV8gI6GSohdv8zl8U=\u003c/HostId\u003e\u003c/Error\u003e"}
y

Yoni Augarten

11/21/2022, 10:26 PM
Are you seeing this error in the lakeFS logs?
Or is this the response you are getting locally from lakeFS?
r

Ronnie Ning

11/21/2022, 10:28 PM
I am using lakefs-client package in python. That was the returned error message
basically, s3 denied lakefs file loading.
y

Yoni Augarten

11/21/2022, 10:30 PM
I see. It means lakeFS doesn't have permissions to write the file. The lakeFS logs can provide more information
But basically you need to take a look at the policy of the IAM user/role that lakeFS operates on behalf of.
r

Ronnie Ning

11/21/2022, 10:33 PM
{
            "Sid": "LakeFSS3",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:Put*",
                "s3:Delete*",
                "s3:Abort*"
            ],
            "Resource": [
                "arn:aws:s3:::uniq-lakefs-dev/*",
                "arn:aws:s3:::uniq-lakefs-dev"
            ]
        },
the policy should be good enough
y

Yoni Augarten

11/21/2022, 10:34 PM
Is this the bucket policy or the policy of the user/role?
r

Ronnie Ning

11/21/2022, 10:34 PM
bucket policy
role policy: full access
y

Yoni Augarten

11/21/2022, 10:35 PM
Please take a look at the logs from the lakeFS server.
r

Ronnie Ning

11/21/2022, 10:36 PM
where is that log in /lakefs ?
y

Yoni Augarten

11/21/2022, 10:36 PM
By default it goes to the standard output.
Of the server process
r

Ronnie Ning

11/21/2022, 10:38 PM
ERROR  [
            2022-11-21T22: 24: 05Z
        ]lakeFS/pkg/block/s3/adapter.go: 239 pkg/block/s3.(*Adapter).streamToS3 bad S3 PutObject response                     error="s3 error: <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>AE8C2RC4F6A8A6HV</RequestId><HostId>Z1CXhP9KOwEUgaijq3Y2TFZDXihy2G9jQDcDeIMZXAm45WrSc859m7jAyZYV8gI6GSohdv8zl8U=</HostId></Error>" host=<http://dvc.rwe.aws.novartis.net|dvc.rwe.aws.novartis.net> method=POST operation=PutObject operation_id=UploadObject path="/api/v1/repositories/data42/branches/test-branch/objects?path=.%2F" request_id=aa3c2df7-e081-49fd-9c34-115cf8658ddf service_name=rest_api status_code=403 url="<https://uniq-lakefs-dev.s3.amazonaws.com/data/gr9ggut0vq9r6bq9np20/cdtvl1d0vq9r6bq9np2g>"
That is the only log.
pretty much the same with the error python lakefs-client returned
y

Yoni Augarten

11/21/2022, 10:44 PM
Right. First thing to verify is that lakeFS is indeed authenticated with the correct role. Can you try to create a new repository, on top of a bucket that doesn't have SSE enabled by default (and does not require it)?
I want to make sure that the problem is indeed related to encryption.
r

Ronnie Ning

11/21/2022, 10:49 PM
the s3 bucket policy also has the following:
{
                    "Sid": "CCDenyUnEncryptedObjectUploads",
                    "Effect": "Deny",
                    "Principal": "*",
                    "Action": "s3:PutObject",
                    "Resource": "arn:aws:s3:::uniq-lakefs-qa/*",
                    "Condition": {
                        "StringNotEquals": {
                            "s3:x-amz-server-side-encryption": "aws:kms"
                        }
                    }
                },
                {
                    "Sid": "CCDenyHttp",
                    "Effect": "Deny",
                    "Principal": "*",
                    "Action": [
                        "s3:GetObject",
                        "s3:PutObject"
                    ],
                    "Resource": "arn:aws:s3:::uniq-lakefs-qa/*",
                    "Condition": {
                        "Bool": {
                            "aws:SecureTransport": "false"
                        }
                    }
                }
just to make sure file is encrypted and using https
lakefs uses https to loading file to s3, right?
y

Yoni Augarten

11/21/2022, 10:51 PM
Yes
Let's first make sure it works on a bucket without any policy.
r

Ronnie Ning

11/21/2022, 10:52 PM
no, I can not change the plocity
y

Yoni Augarten

11/21/2022, 10:53 PM
I mean, create a new bucket just for the experiment.
r

Ronnie Ning

11/21/2022, 10:54 PM
the bucket policy is automatically generated when creating a s3 bucket and I can not change it
y

Yoni Augarten

11/21/2022, 10:54 PM
I see. In that case please open an issue and I will try to reproduce the scenario tomorrow.
r

Ronnie Ning

11/21/2022, 10:55 PM
great, I will do it tomorrow.
👍🏻 1
I already opened an issue: https://github.com/treeverse/lakeFS/issues/4635
👍🏻 1
View count: 14
Back to #lakefs-for-beginnersJoin thread in Slack