https://lakefs.io/ logo
Join Slack
Powered by
Hey all, I’ve been able to get everything deployed...
# help
t
Hey all, I’ve been able to get everything deployed to our cluster and get the initial user created. Now I’m running into issues creating the initial repository. I get 
EmptyStaticCreds: static credentials are empty
even though I’ve verified the keys are in the config file mounted in the pod. Scrubbed logs in more details in this thread. Has anyone ran into this issue and have a fix?
Copy code
Using config file: /etc/lakefs/config.yaml
time="2021-05-03T21:39:05Z" level=info msg="lakeFS run" func="pkg/logging.(*logrusEntryWrapper).Infof" file="build/pkg/logging/logger.go:115" version=0.33.1
time="2021-05-03T21:39:05Z" level=info msg="connecting to the DB" func=pkg/db.ConnectDBPool file="build/pkg/db/connect.go:54" conn_max_lifetime=5m0s driver=pgx max_idle_conns=25 max_open_conns=25 uri="<postgres://user:pass@aif-lakefs-db.dataops.svc.cluster.local:5432/lakefs?sslmode=disable|postgres://user:pass@aif-lakefs-db.dataops.svc.cluster.local:5432/lakefs?sslmode=disable>"
time="2021-05-03T21:39:05Z" level=info msg="initialized DB connection" func=pkg/db.ConnectDBPool file="build/pkg/db/connect.go:73" conn_max_lifetime=5m0s driver=pgx max_idle_conns=25 max_open_conns=25 uri="<postgres://user:pass@aif-lakefs-db.dataops.svc.cluster.local:5432/lakefs?sslmode=disable|postgres://user:pass@aif-lakefs-db.dataops.svc.cluster.local:5432/lakefs?sslmode=disable>"
time="2021-05-03T21:39:05Z" level=info msg="connecting to the DB" func=pkg/db.ConnectDBPool file="build/pkg/db/connect.go:54" conn_max_lifetime=5m0s driver=pgx max_idle_conns=25 max_open_conns=25 uri="<postgres://user:pass@aif-lakefs-db.dataops.svc.cluster.local:5432/lakefs?sslmode=disable|postgres://user:pass@aif-lakefs-db.dataops.svc.cluster.local:5432/lakefs?sslmode=disable>"
time="2021-05-03T21:39:05Z" level=info msg="initialized DB connection" func=pkg/db.ConnectDBPool file="build/pkg/db/connect.go:73" conn_max_lifetime=5m0s driver=pgx max_idle_conns=25 max_open_conns=25 uri="<postgres://user:pass@aif-lakefs-db.dataops.svc.cluster.local:5432/lakefs?sslmode=disable|postgres://user:pass@aif-lakefs-db.dataops.svc.cluster.local:5432/lakefs?sslmode=disable>"
time="2021-05-03T21:39:05Z" level=info msg="connecting to the DB" func=pkg/db.ConnectDBPool file="build/pkg/db/connect.go:54" conn_max_lifetime=5m0s driver=pgx max_idle_conns=25 max_open_conns=25 uri="<postgres://user:pass@aif-lakefs-db.dataops.svc.cluster.local:5432/lakefs?sslmode=disable|postgres://user:pass@aif-lakefs-db.dataops.svc.cluster.local:5432/lakefs?sslmode=disable>"
time="2021-05-03T21:39:05Z" level=info msg="initialized DB connection" func=pkg/db.ConnectDBPool file="build/pkg/db/connect.go:73" conn_max_lifetime=5m0s driver=pgx max_idle_conns=25 max_open_conns=25 uri="<postgres://user:pass@aif-lakefs-db.dataops.svc.cluster.local:5432/lakefs?sslmode=disable|postgres://user:pass@aif-lakefs-db.dataops.svc.cluster.local:5432/lakefs?sslmode=disable>"
time="2021-05-03T21:39:05Z" level=info msg="initialize blockstore adapter" func=pkg/block/factory.BuildBlockAdapter file="build/pkg/block/factory/build.go:38" type=s3
time="2021-05-03T21:39:05Z" level=info msg="initialized blockstore adapter" func=pkg/block/factory.buildS3Adapter file="build/pkg/block/factory/build.go:97" type=s3
time="2021-05-03T21:39:05Z" level=info msg="initialize blockstore adapter" func=pkg/block/factory.BuildBlockAdapter file="build/pkg/block/factory/build.go:38" type=s3
time="2021-05-03T21:39:05Z" level=info msg="initialized blockstore adapter" func=pkg/block/factory.buildS3Adapter file="build/pkg/block/factory/build.go:97" type=s3
time="2021-05-03T21:39:05Z" level=info msg="initialized Auth service" func=pkg/auth.NewDBAuthService file="build/pkg/auth/service.go:180"
time="2021-05-03T21:39:05Z" level=warning msg="EmptyStaticCreds: static credentials are empty: failed to get AWS account ID for BI" func="pkg/logging.(*logrusEntryWrapper).Warnf" file="build/pkg/logging/logger.go:119"
time="2021-05-03T21:39:05Z" level=info msg="initialize OpenAPI server" func=pkg/api.Serve file="build/pkg/api/serve.go:42" service=api_gateway
time="2021-05-03T21:39:06Z" level=info msg="initialized S3 Gateway handler" func=pkg/gateway.NewHandler file="build/pkg/gateway/handler.go:121" s3_bare_domain=<http://lakefs-s3.xxxx.xxxx.xxx.xxxx.xxx|lakefs-s3.xxxx.xxxx.xxx.xxxx.xxx> s3_region=us-east-1
time="2021-05-03T21:39:06Z" level=info msg="starting HTTP server" func=cmd/lakefs/cmd.glob..func9 file="cmd/run.go:158" listen_address="0.0.0.0:8000"
time="2021-05-03T21:39:06Z" level=info msg="Up and running (^C to shutdown)..." func=cmd/lakefs/cmd.gracefulShutdown file="cmd/run.go:220" version=0.33.1


    ██╗      █████╗ ██╗  ██╗███████╗███████╗███████╗
    ██║     ██╔══██╗██║ ██╔╝██╔════╝██╔════╝██╔════╝
    ██║     ███████║█████╔╝ █████╗  █████╗  ███████╗
    ██║     ██╔══██║██╔═██╗ ██╔══╝  ██╔══╝  ╚════██║
    ███████╗██║  ██║██║  ██╗███████╗██║     ███████║
    ╚══════╝╚═╝  ╚═╝╚═╝  ╚═╝╚══════╝╚═╝     ╚══════╝

│
│ If you're running lakeFS locally for the first time,
│     complete the setup process at <http://127.0.0.1:8000/setup|http://127.0.0.1:8000/setup>
│

│
│ For more information on how to use lakeFS,
│     check out the docs at <https://docs.lakefs.io/quickstart/repository|https://docs.lakefs.io/quickstart/repository>
│

Version 0.33.1

time="2021-05-03T21:39:44Z" level=debug msg="rows done" func="pkg/db.(*LoggedRows).logDuration" file="build/pkg/db/logged_rows.go:34" args="[dataops ]" duration=3.082832ms query="\n\t    WITH resolved_policies_view AS (\n                SELECT <http://auth_policies.id|auth_policies.id>, auth_policies.created_at, auth_policies.display_name, auth_policies.statement, auth_users.display_name AS user_display_name\n                FROM auth_policies INNER JOIN\n                     auth_user_policies ON (<http://auth_policies.id|auth_policies.id> = auth_user_policies.policy_id) INNER JOIN\n\t\t     auth_users ON (<http://auth_users.id|auth_users.id> = auth_user_policies.user_id)\n                UNION\n\t\tSELECT <http://auth_policies.id|auth_policies.id>, auth_policies.created_at, auth_policies.display_name, auth_policies.statement, auth_users.display_name AS user_display_name\n\t\tFROM auth_policies INNER JOIN\n\t\t     auth_group_policies ON (<http://auth_policies.id|auth_policies.id> = auth_group_policies.policy_id) INNER JOIN\n\t\t     auth_groups ON (<http://auth_groups.id|auth_groups.id> = auth_group_policies.group_id) INNER JOIN\n\t\t     auth_user_gro...
time="2021-05-03T21:39:44Z" level=debug msg="performing API action" func="pkg/api.(*Controller).LogAction" file="build/pkg/api/controller.go:105" action=list_repos host=<http://lakefs.xxxx.xxxx.xxx.xxxx.com|lakefs.xxxx.xxxx.xxx.xxxx.com> message_type=action method=GET path="/api/v1/repositories?after=&amount=300" request_id=bfb89f1b-a036-4d96-8c83-eb5ce7645b2f service=api_gateway service_name=rest_api user=dataops
time="2021-05-03T21:39:44Z" level=debug msg="performing API action" func="pkg/api.(*Controller).LogAction" file="build/pkg/api/controller.go:105" action=create_repo host=<http://lakefs.xxxx.xxxx.xxx.xxxx.xxx|lakefs.xxxx.xxxx.xxx.xxxx.xxx> message_type=action method=POST path=/api/v1/repositories request_id=2d01c128-0339-4fef-8828-29dbe0fbbd17 service=api_gateway service_name=rest_api user=dataops
time="2021-05-03T21:39:44Z" level=error msg="failed to sign request" func="pkg/block/s3.(*Adapter).streamToS3" file="build/pkg/block/s3/adapter.go:174" error="EmptyStaticCreds: static credentials are empty" host=lakefs. <http://xxxx.xxxx.xxx.xxxx.xxx|xxxx.xxxx.xxx.xxxx.xxx> method=POST operation=PutObject path=/api/v1/repositories request_id=2d01c128-0339-4fef-8828-29dbe0fbbd17 service_name=rest_api user=dataops
time="2021-05-03T21:39:44Z" level=warning msg="Could not access storage namespace" func="pkg/api.(*Controller).CreateRepositoryHandler.func1" file="build/pkg/api/controller.go:567" error="EmptyStaticCreds: static credentials are empty" service=api_gateway storage_namespace="<s3://lakefs-store-43562dfe-e06b-4b9b-bf75-d0d0f6c8d1bb/|s3://lakefs-store-43562dfe-e06b-4b9b-bf75-d0d0f6c8d1bb/>"
time="2021-05-03T21:39:44Z" level=debug msg="HTTP call ended" func=pkg/httputil.DebugLoggingMiddleware.func1 file="build/pkg/httputil/logging.go:77" host=<http://lakefs.xxxx.xxxx.xxx.xxxx.xxx|lakefs.xxxx.xxxx.xxx.xxxx.xxx> method=POST path=/api/v1/repositories request_id=2d01c128-0339-4fef-8828-29dbe0fbbd17 sent_bytes=76 service_name=rest_api status_code=400 took=11.372304ms user=dataops
time="2021-05-03T21:39:44Z" level=debug msg="rows done" func="pkg/db.(*LoggedRows).logDuration" file="build/pkg/db/logged_rows.go:34" args="[ 1000]" duration="699.955µs" query="\n\t\t\tSELECT id, storage_namespace, creation_date, default_branch\n\t\t\tFROM graveler_repositories\n\t\t\tWHERE id >= $1\n\t\t\tORDER BY id ASC\n\t\t\tLIMIT $2" type="start query"
time="2021-05-03T21:39:44Z" level=debug msg="HTTP call ended" func=pkg/httputil.DebugLoggingMiddleware.func1 file="build/pkg/httputil/logging.go:77" host=lakefs. <http://xxxx.xxxx.xxx.xxxx.xxx|xxxx.xxxx.xxx.xxxx.xxx> method=GET path="/api/v1/repositories?after=&amount=300" request_id=bfb89f1b-a036-4d96-8c83-eb5ce7645b2f sent_bytes=79 service_name=rest_api status_code=200 took=11.944864ms user=dataops
time="2021-05-03T21:39:44Z" level=debug msg="performing API action" func="pkg/api.(*Controller).LogAction" file="build/pkg/api/controller.go:105" action=list_repos host=lakefs. <http://xxxx.xxxx.xxx.xxxx.xxx|xxxx.xxxx.xxx.xxxx.xxx> message_type=action method=GET path="/api/v1/repositories?after=&amount=300" request_id=ccc12af9-7f8b-464a-bc21-86e693201e4c service=api_gateway service_name=rest_api user=dataops
time="2021-05-03T21:39:44Z" level=debug msg="rows done" func="pkg/db.(*LoggedRows).logDuration" file="build/pkg/db/logged_rows.go:34" args="[ 1000]" duration="522.882µs" query="\n\t\t\tSELECT id, storage_namespace, creation_date, default_branch\n\t\t\tFROM graveler_repositories\n\t\t\tWHERE id >= $1\n\t\t\tORDER BY id ASC\n\t\t\tLIMIT $2" type="start query"
time="2021-05-03T21:39:44Z" level=debug msg="HTTP call ended" func=pkg/httputil.DebugLoggingMiddleware.func1 file="build/pkg/httputil/logging.go:77" host=lakefs. <http://xxxx.xxxx.xxx.xxxx.xxx|xxxx.xxxx.xxx.xxxx.xxx> method=GET path="/api/v1/repositories?after=&amount=300" request_id=ccc12af9-7f8b-464a-bc21-86e693201e4c sent_bytes=79 service_name=rest_api status_code=200 took="822.009µs" user=dataops
For my deployment I’m leveraging an OpenShift cluster and using their 
ObjectBucketClaims
for the underlying S3 bucket. We’re also behind a corporate proxy but I’ve already tried setting the environmental variables for that as well. LakeFS logs:
Also, I’ve verified the config file mounted to the pod has the correct keys:
Copy code
/home/lakefs $ ls -ltrh /etc/lakefs/
total 0
lrwxrwxrwx    1 root     root          18 May  3 21:39 config.yaml -> ..data/config.yaml
/home/lakefs $ cat /etc/lakefs/config.yaml
stats.enabled: false
logging.level: DEBUG
blockstore:
 type: s3
 s3:
   region: us-east-1
   endpoint: <https://s3-openshift-storage.xxxx.xxx.xxx.xxxx.xxx:443|https://s3-openshift-storage.xxxx.xxx.xxx.xxxx.xxx:443>
   credentials:
     access_key_id: accesskey
     secret_access_key: secretkey

gateways:
 s3:
   domain_name: <http://lakefs-s3.xxxx.xxx.xxx.xxxx.xxx|lakefs-s3.xxxx.xxx.xxx.xxxx.xxx>
   region: us-east-1
/home/lakefs $
o
Hey! i’m assuming this is an on-prem deployment with an s3 compatible object store?
when a repository is created lakeFS attempts to write an object to the specified storage namespace and read it back - seems like the AWS sdk is returning an error while trying to do so.
t
Hey Oz! Yes it’s an on-prem cluster and using OpenShifts S3 object storage
Yea I tried tracking that error down and it just seems to be thrown when the credentials are empty https://github.com/aws/aws-sdk-go/blob/c5aeec1660a2d951fcf0fadf27a03f6eb4baf8d2/aws/credentials/static_provider.go#L42. It also has a similar error in the LakeFS logs at startup doing something with logging
If it helps I’ve also tried setting the credentials as environmental variables and had the same logs
o
yes it seems that for some reason the credentials arrive empty when initializing the AWS SDK
Thanks for sharing the logs and config file! i’m going to open an issue and try to reproduce with a similar setup.
🙌 1
t
No problem! Let me know if you need any other information
👍 1
o
Hey @Thomas Vander Wal - I've updated the issue but I'll paste the solution here too for reference: Root cause found: Since we don't version our documentation (yet!), the current docs specify 
blockstore.s3.credentials.secret_access_key
- this is true since version 0.40.0. For prior version the setting was called 
blockstore.s3.credentials.access_secret_key
. I recommend either upgrading from 0.33.1 to current latest (0.40.3 at the time of writing). Alternatively, if there's good reason to stick with 0.33.1 for now - replace the 
secret_access_key
directive with 
access_secret_key
.
In any case - good catch 🙂 and sorry for the inconvenience - There's an open issue for versioning of the documentation itself - you may track it (and of course comment and collaborate) here: https://github.com/treeverse/lakeFS/issues/1612
t
@Oz Katz awesome I’ll upgrade today between meetings. Thanks for digging into that.
🙏 1
o
My pleasure! let me know how it went and if there’s anything else I can help with.
t
That did get me passed that issue! Though now I’m running into issues with our Self Signed Certificate. Going to spend some time today and see if I can’t get that injected and configured in the pod.
🤘 1
o
Ah got it. Not sure what the idiomatic Kubernetes approach for this is - Would love to understand what lakeFS can do to make that easier. For context, the lakeFS image is built on Alpine (see https://github.com/treeverse/lakeFS/blob/master/Dockerfile#L1) so something along these lines would probably work but this means creating your own little image that extends the default lakeFS one.
t
Thanks for the info! Yea in the past we’ve generally extended an image, added certs/proxy, then host it. But that makes upgrading a pain. I wonder if it wouldn’t be better to mount the certificate file as a secret and the able to set 
SSL_CERT_FILE
or whatever variable the go SDKs respect in the values file.
o
@Thomas Vander Wal good idea! didn't think about that route. From the looks of it, https://docs.aws.amazon.com/sdk-for-go/api/aws/session/ - there's a 
$AWS_CA_BUNDLE
env var which sounds very relevant for this. Would be happy to know if that works for you (and if it does we can add it to the official docs as well)
t
@Oz Katz mounting the secret to 
/etc/ssl/certs
did the trick and didn’t need to set the environmental variable
👍 2
o
great! happy to hear that.
Open in Slack
PreviousNext
Powered by