I have LakeFS docker container working fine with s3 and Postgres. When I switch from Postgres to DynamoDB, it seems to connect to DynamoDB service but CreateTable fails with tls:Failed to verify certificate... Certificate signed by unknown authority. I don't see any TLS related config settings in https://docs.lakefs.io/reference/configuration.html#example-aws-deployment-with-dynamodb-database. How do we get past this?

Hi @Sam, Can you please post the full lakeFS logs. Also, where are you running the docker container from?

Running the LakeFS docker image from linux command line - docker run .... It may not be related to LakeFS configuration... I get same cert error when I run the aws cli command - aws dynamodb list-tables. But when I set the AWS_CA_BUNDLE to point to the certs location, the CLI command works fine. Is there a lakefs config.yaml config parameter to specify AWS_CA_BUNDLE ?

No, but if I'm not mistaken if you export this env var in the container it should work as well

Thanks... When I pass the AWS_CA_BUNDLE env variable, it still fails with - {"error":"not found","file":"build/pkg/stats/metadata.go:34", "func":pkg/stats.NewMetaData"... "failed to collect account metadata"} "error":"operation error STS:GetCallerIdentity, exceeded max number of attempts.... x509 certificate signed by unknown authority... When I try from CLI "aws sts get-caller-identity " after setting AWS_CA_BUNDLE env variable to point to ca-bundle.pem, it works fine. However passing the same AWS_CA_BUNDLE env variable into the docker run lakefs produces the errors above. Probably something simple... any ideas? Or as a temporary workaround... any way to tell it --no-verify-ssl ? I got this lakefs docker image several months ago... not sure what version it is.. What is the latest stable version #?

You can find the lakeFS releases here. Please try to update, it might help with your issue

ok, thanks.. How do I tell what version I'm running? I don't recall seeing anything in the trace . I will download a newer version tomorrow and see if that helps.

It's printed out in the lakeFS logs on startup, you can also see it in the top right corner in the UI if you click on your username

Does the container use aws cli or sdk? When I try any aws command before the container fails and exits, e.g. - "*docker exec lakefs aws sts get-caller-identity*" it fails with unable to find aws executable. Do any other aws env variables need to be set in config.yaml besides aws region? When I run - "*docker exec lakefs env*" before the container fails and exits, I can see the env var AWS_CA_BUNDLE correctly pointing to aws-ca-bundle.pem although the call to https://sts.xxx.yyy still fails with cert signed by unknown authority. I'll probably give up and try downloading latest version of lakefs.

When we initialize the KV driver and Storage adapter for AWS we use the AWS SDK chain authentication which should also try to load AWS env vars if exist. I suggest you try to upgrade the lakeFS version and if it still happens there we can take a closer look

ok thanks

Hi Sam, You can use either one. I usually run in K8s or Docker, so naturally I prefer to pull an image from Docker. If you intend to run a binary, a release is probably more appropriate.

I downloaded the latest release 1.29.0. The STS issue is gone. All good! However, when I access lakefs from CLI with the command (I have lakefs behind NGINX for https) -- aws s3 --profile lakefs --endpoint-url https://lakefs.mydomain.com ls s3://myrepo/main/foobar I get the following error - SSL validation failed for https://lakefs.mydomain.com/myrepo?list-type=2&prefix=main...&encoding-type=url {SSL: CERTIFICATE_VERIFY_FAILED]... unable to get local issuer certificate (_ssl.c:1006) If I access lakefs with CLI on HTTP it works fine - http://lakefs.mydomain.com:8000 ls s3://myrepo/main/foobar -which clearly points to a cert issue, but the CLI command aws s3 ls s3://mybucket works fine when I set AWS_CA_BUNDLE to point to the the right aws ca-bundle.pem. In addition I also set ca_bundle in ~/.aws/config for lakefs profile but https to lakefs still fails with unable to get local issuer certificate (_ssl.c:1006). I've had the above problem both with the older lakefs container version and also the latest binary release 1.29.0.

Hi, Glad to hear you're making progress! It seems that your current problems are between the aws cli and the nginx reverse proxy. There are many details, please let me try to resummarize so that we can try to make useful suggestions: • The aws cli works with AWS S3 if you set the environment variable AWS_CA_BUNDLE. • It fails with your nginx if you set ca_bundle in the config file. Is this accurate?