Title
#lakefs-for-beginners
r

Ronnie Ning

11/16/2022, 6:28 PM
Anyone knows how to set config.yaml for aws dynamodb? I used the example on lakefs for deploy on aws, I got
ValidationException: The provided key element does not match the schema
Itai Admi

Itai Admi

11/16/2022, 6:33 PM
Hey Ronnie, I'll try to help with this. Are you getting this error from lakeFS? Can you share the logs (without truncation)?
r

Ronnie Ning

11/16/2022, 6:44 PM
Sorry, just solved this issue. Solution is that we don't need to create a dynamodb table first before we start lakefs. Lakefs will automatically create one for you as long as it has the right role. Thanks Itai
Itai Admi

Itai Admi

11/16/2022, 6:46 PM
🙏 Let me know if you need anything else
r

Ronnie Ning

11/16/2022, 8:45 PM
BTW, the folder in S3 bucket generated by LakeFS is dummy by default. Is there a way to customize it?
Itai Admi

Itai Admi

11/16/2022, 8:49 PM
What do you mean by “dummy by default”?
r

Ronnie Ning

11/16/2022, 8:55 PM
Folder name in the bucket is called
dummy
Itai Admi

Itai Admi

11/16/2022, 9:07 PM
dummy
is an object lakeFS creates inside the storage namespace to validate it has access.
r

Ronnie Ning

11/17/2022, 3:13 PM
Do you know what kind of permissions I have to have in order to make lakefs work? I set s3 full access, kms, and dynamodb full access, but I still fail to upload files in lakefs.
Itai Admi

Itai Admi

11/17/2022, 3:16 PM
What is the error you're getting? Did you successfully create a repo? I find this doc very useful for storage permissions
r

Ronnie Ning

11/17/2022, 3:17 PM
I can create repo, branch, but fail to upload files
Itai Admi

Itai Admi

11/17/2022, 3:18 PM
That's wierd, can you share the logs?
r

Ronnie Ning

11/17/2022, 3:20 PM
how do we use kms when uploading a file?
3:22 PM
in gui, there is no way to use kms, right? We can only use code, for example, lakefs-python package to upload files after kms, right?
Itai Admi

Itai Admi

11/17/2022, 3:22 PM
What's kms? 😅
r

Ronnie Ning

11/17/2022, 3:23 PM
3:23 PM
to encrypt or decrypt data
Itai Admi

Itai Admi

11/17/2022, 3:25 PM
In s3 you can configure bucket level encryption, every object stored in the bucket will seamlessly be stored encrypted. You don't need to do it yourself before storing the data in lakeFS.
r

Ronnie Ning

11/17/2022, 3:28 PM
yeah, already enabled default encryption
3:30 PM
in our s3 bucket policy, there is one manditory rule, data has to be encrypted using kms before uploading. Maybe that's why my uploading was failed. So I can not use gui to upload data, but have to do it in code.
Itai Admi

Itai Admi

11/17/2022, 3:32 PM
The UI enables you to upload data from your local storage, so unless you have the encrypted files stored locally, I guess you're right.
r

Ronnie Ning

11/17/2022, 3:33 PM
Thanks
Itai Admi

Itai Admi

11/17/2022, 3:34 PM
🙏
r

Ronnie Ning

11/21/2022, 9:45 PM
There is no way to have some kind of kms setup in lakefs, right?
Yoni Augarten

Yoni Augarten

11/21/2022, 9:53 PM
Hey @Ronnie Ning, I'm not sure what you're asking. If you're asking whether lakeFS can encrypt your objects on S3 using KMS with server-side encryption, then it's currently not supported.
r

Ronnie Ning

11/21/2022, 10:03 PM
I tried several ways to encrypt file, then uploaded it to s3 which is enabled sse, all were failed. But if I put kms as an extra config, it was successful. So I wonder if lakefs can also set kms as config when loading files to s3.
Yoni Augarten

Yoni Augarten

11/21/2022, 10:10 PM
Unfortunately, this is not supported at the moment. You're welcome to open an issue describing the use case.
r

Ronnie Ning

11/21/2022, 10:13 PM
Iddo Avneri 10/26/2022, 12:18 PM You are welcome to run lakeFS on top of a bucket with default SSE - it will work and all the data will be encrypted.
Yoni Augarten

Yoni Augarten

11/21/2022, 10:14 PM
I wasn't aware of the option to turn on SSE by default. Let me check.
r

Ronnie Ning

11/21/2022, 10:14 PM
I wonder how since I followed the instructions :1. Prepare your encrypted storage. 2. Run lakeFS locally against your storage, but no lucky
10:15 PM
Following these instructions will cause all new objects on your bucket to be encrypted by default. I hope this helps.
r

Ronnie Ning

11/21/2022, 10:17 PM
Yeah, I did enable the default sse on s3, but lakefs file loading failed with access denied.
Yoni Augarten

Yoni Augarten

11/21/2022, 10:18 PM
Please provide the complete error that you are seeing
r

Ronnie Ning

11/21/2022, 10:24 PM
HTTP response headers: HTTPHeaderDict({'Date': 'Mon, 21 Nov 2022 22:24:05 GMT', 'Content-Type': 'application/json', 'Content-Length': '383', 'Connection': 'keep-alive', 'X-Content-Type-Options': 'nosniff', 'X-Request-Id': 'aa3c2df7-e081-49fd-9c34-115cf8658ddf'})
HTTP response body: {"message":"s3 error: \u003c?xml version=\"1.0\" encoding=\"UTF-8\"?\u003e\n\u003cError\u003e\u003cCode\u003eAccessDenied\u003c/Code\u003e\u003cMessage\u003eAccess Denied\u003c/Message\u003e\u003cRequestId\u003eAE8C2RC4F6A8A6HV\u003c/RequestId\u003e\u003cHostId\u003eZ1CXhP9KOwEUgaijq3Y2TFZDXihy2G9jQDcDeIMZXAm45WrSc859m7jAyZYV8gI6GSohdv8zl8U=\u003c/HostId\u003e\u003c/Error\u003e"}
Yoni Augarten

Yoni Augarten

11/21/2022, 10:26 PM
Are you seeing this error in the lakeFS logs?
10:27 PM
Or is this the response you are getting locally from lakeFS?
r

Ronnie Ning

11/21/2022, 10:28 PM
I am using lakefs-client package in python. That was the returned error message
10:29 PM
basically, s3 denied lakefs file loading.
Yoni Augarten

Yoni Augarten

11/21/2022, 10:30 PM
I see. It means lakeFS doesn't have permissions to write the file. The lakeFS logs can provide more information
10:31 PM
But basically you need to take a look at the policy of the IAM user/role that lakeFS operates on behalf of.
r

Ronnie Ning

11/21/2022, 10:33 PM
{
            "Sid": "LakeFSS3",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "s3:Get*",
                "s3:List*",
                "s3:Put*",
                "s3:Delete*",
                "s3:Abort*"
            ],
            "Resource": [
                "arn:aws:s3:::uniq-lakefs-dev/*",
                "arn:aws:s3:::uniq-lakefs-dev"
            ]
        },
10:33 PM
the policy should be good enough
Yoni Augarten

Yoni Augarten

11/21/2022, 10:34 PM
Is this the bucket policy or the policy of the user/role?
r

Ronnie Ning

11/21/2022, 10:34 PM
bucket policy
10:34 PM
role policy: full access
Yoni Augarten

Yoni Augarten

11/21/2022, 10:35 PM
Please take a look at the logs from the lakeFS server.
r

Ronnie Ning

11/21/2022, 10:36 PM
where is that log in /lakefs ?
Yoni Augarten

Yoni Augarten

11/21/2022, 10:36 PM
By default it goes to the standard output.
10:37 PM
Of the server process
r

Ronnie Ning

11/21/2022, 10:38 PM
ERROR  [
            2022-11-21T22: 24: 05Z
        ]lakeFS/pkg/block/s3/adapter.go: 239 pkg/block/s3.(*Adapter).streamToS3 bad S3 PutObject response                     error="s3 error: <?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>AE8C2RC4F6A8A6HV</RequestId><HostId>Z1CXhP9KOwEUgaijq3Y2TFZDXihy2G9jQDcDeIMZXAm45WrSc859m7jAyZYV8gI6GSohdv8zl8U=</HostId></Error>" host=<http://dvc.rwe.aws.novartis.net|dvc.rwe.aws.novartis.net> method=POST operation=PutObject operation_id=UploadObject path="/api/v1/repositories/data42/branches/test-branch/objects?path=.%2F" request_id=aa3c2df7-e081-49fd-9c34-115cf8658ddf service_name=rest_api status_code=403 url="<https://uniq-lakefs-dev.s3.amazonaws.com/data/gr9ggut0vq9r6bq9np20/cdtvl1d0vq9r6bq9np2g>"
10:39 PM
That is the only log.
10:39 PM
pretty much the same with the error python lakefs-client returned
Yoni Augarten

Yoni Augarten

11/21/2022, 10:44 PM
Right. First thing to verify is that lakeFS is indeed authenticated with the correct role. Can you try to create a new repository, on top of a bucket that doesn't have SSE enabled by default (and does not require it)?
10:45 PM
I want to make sure that the problem is indeed related to encryption.
r

Ronnie Ning

11/21/2022, 10:49 PM
the s3 bucket policy also has the following:
10:49 PM
{
                    "Sid": "CCDenyUnEncryptedObjectUploads",
                    "Effect": "Deny",
                    "Principal": "*",
                    "Action": "s3:PutObject",
                    "Resource": "arn:aws:s3:::uniq-lakefs-qa/*",
                    "Condition": {
                        "StringNotEquals": {
                            "s3:x-amz-server-side-encryption": "aws:kms"
                        }
                    }
                },
                {
                    "Sid": "CCDenyHttp",
                    "Effect": "Deny",
                    "Principal": "*",
                    "Action": [
                        "s3:GetObject",
                        "s3:PutObject"
                    ],
                    "Resource": "arn:aws:s3:::uniq-lakefs-qa/*",
                    "Condition": {
                        "Bool": {
                            "aws:SecureTransport": "false"
                        }
                    }
                }
10:50 PM
just to make sure file is encrypted and using https
10:51 PM
lakefs uses https to loading file to s3, right?
Yoni Augarten

Yoni Augarten

11/21/2022, 10:51 PM
Yes
10:52 PM
Let's first make sure it works on a bucket without any policy.
r

Ronnie Ning

11/21/2022, 10:52 PM
no, I can not change the plocity
Yoni Augarten

Yoni Augarten

11/21/2022, 10:53 PM
I mean, create a new bucket just for the experiment.
r

Ronnie Ning

11/21/2022, 10:54 PM
the bucket policy is automatically generated when creating a s3 bucket and I can not change it
Yoni Augarten

Yoni Augarten

11/21/2022, 10:54 PM
I see. In that case please open an issue and I will try to reproduce the scenario tomorrow.
r

Ronnie Ning

11/21/2022, 10:55 PM
great, I will do it tomorrow.