Hi team I have one question about using service account to d

Question

Hi team I have one question about using service account to deploy service To use S3 as blockstore before my helm config file is like below ```secrets authEncryptSecretKey 123 Use the following to fetch PostgreSQL connection string from an existing secret databaseConnectionString postgres lakefsConfig | database type postgres blockstore type s3 s3 region us west 2 credentials access key id secret access key ``` If I want to switch to service account to access S3 bucket do you have any example for the helm config files Will it be something like below ```serviceAccount lt service account name gt secrets authEncryptSecretKey 123 Use the following to fetch PostgreSQL connection string from an existing secret databaseConnectionString postgres lakefsConfig | database type postgres blockstore type s3 s3 region us west 2 ```

Amit Kesarwani · Answer

lakeFS support using credentials file using the `blockstore s3 credentials file` and `blockstore s3 profile`configurations You can give a path to a configuration file that will look something like that ``` lakefs role arn = lt YOUR ROLE ARN gt web identity token file = var run secrets eks amazonaws com serviceaccount token role session name = lt ROLE SESSION NAME gt ```

gang ye · Answer

Thanks for the quick reply I will have a try to use profile But serviceAccount is the preferred way for us I saw the helm values can take serviceAccount value I guess that s supported So trying to get some example to follow

Amit Kesarwani · Answer

< gang ye> FYI I requested engineers to review this request and provide some example to use service account

gang ye · Answer

OK Thanks a lot

Itai Admi · Answer

< gang ye> the lakeFS server uses the golang aws sdk If you configured the service account for the lakeFS pod deployment properly it should work out of the box So this config ```secrets authEncryptSecretKey 123 Use the following to fetch PostgreSQL connection string from an existing secret databaseConnectionString postgres lakefsConfig | database type postgres blockstore type s3 s3 region us west 2 ``` will default to aws sdk credentials lookup which will find the service account creds

gang ye · Answer

I think I figured out how to config the service account in helm config file like below ```serviceAccount name lt service account name gt ```

gang ye · Answer

one quick question does lakefs uses AWS `DefaultAWSCredentialsProviderChain` after setting service account in helm deployment pod env variables have the configuration below ```AWS DEFAULT REGION=us west 2 AWS REGION=us west 2 AWS ROLE ARN=arn aws iam role data experimentation AWS WEB IDENTITY TOKEN FILE= var run secrets eks amazonaws com serviceaccount token AWS STS REGIONAL ENDPOINTS=regional``` But the lakefs server cannot create the block adapter as expected

gang ye · Answer

Hi < Itai Admi> I was able to set the service account in deployment The pod contains the above env variable And I verified the service account credential works by running another pod with awscli to list files on S3 But still lakefs server cannot initialize the block adapter Any other missing configuration Do you have any clue Thanks

Itai Admi · Answer

Hey < gang ye> can you share the lakeFS config that you re using now including the env vars for it if any starting with `LAKEFS ` Also can you share the logs from the server

gang ye · Answer

Sure Helm config file ```image repository <http docker io treeverse lakefs|docker io treeverse lakefs> pullPolicy IfNotPresent Keys used for existingSecret secrets authEncryptSecretKey 123 lakefsConfig | logging level TRACE stats enabled false database type local blockstore type s3 serviceAccount name data experimentation sa``` env variable ```~ $ printenv | grep lakefs HOSTNAME=lakefsingress cc96645cf 46xs7 HOME= home lakefs PWD= home lakefs ~ $ printenv | grep AWS AWS ROLE ARN=arn aws iam role data experimentation AWS WEB IDENTITY TOKEN FILE= var run secrets eks amazonaws com serviceaccount token AWS STS REGIONAL ENDPOINTS=regional AWS DEFAULT REGION=us west 2 AWS REGION=us west 2```

gang ye · Answer

server pod log message ```time= 2023 10 20T21 08 50Z level=info msg= Configuration file func=github com treeverse lakefs cmd lakefs cmd initConfig file= build cmd lakefs cmd root go 109 fields file= etc lakefs config yaml file= build cmd lakefs cmd root go 109 phase=startup time= 2023 10 20T21 08 50Z level=info msg= Config loaded func=cmd lakefs cmd initConfig file= cmd root go 151 fields file= etc lakefs config yaml file= cmd root go 151 phase=startup time= 2023 10 20T21 08 50Z level=info msg=Config func=cmd lakefs cmd initConfig file= cmd root go 159 actions enabled=true actions lua net http enabled=false auth api endpoint= auth api supports invites=false auth api token= auth cache enabled=true auth cache jitter=3s auth cache size=1024 auth cache ttl=20s auth cookie auth verification auth source= auth cookie auth verification default initial groups= auth cookie auth verification external user id claim name= auth cookie auth verification friendly name claim name= auth cookie auth verification initial groups claim name= auth cookie auth verification validate id token claims= map auth encrypt secret key= auth login duration=168h0m0s auth logout redirect url= auth login auth oidc default initial groups= auth oidc friendly name claim name= auth oidc initial groups claim name= auth oidc validate id token claims= map auth remote authenticator default user group=Viewers auth remote authenticator enabled=false auth remote authenticator endpoint= auth remote authenticator request timeout=10s auth ui config login cookie names= internal auth session auth ui config login failed message= The credentials don t match auth ui config login url= auth ui config logout url= auth ui config rbac=simplified blockstore azure auth method= blockstore azure disable pre signed=false blockstore azure disable pre signed ui=true blockstore azure pre signed expiry=15m0s blockstore azure storage access key= blockstore azure storage account= blockstore azure test endpoint url= blockstore azure try timeout=10m0s blockstore gs credentials file= blockstore gs credentials json= blockstore gs disable pre signed=false blockstore gs disable pre signed ui=true blockstore gs pre signed expiry=15m0s blockstore gs s3 endpoint= blockstore local allowed external prefixes= blockstore local import enabled=false blockstore local import hidden=false blockstore local path= ~ lakefs data block blockstore s3 client log request=false blockstore s3 client log retries=false blockstore s3 credentials file= blockstore s3 disable pre signed=false blockstore s3 disable pre signed ui=true blockstore s3 discover bucket region=true blockstore s3 endpoint= blockstore s3 force path style=false blockstore s3 max retries=5 blockstore s3 pre signed expiry=15m0s blockstore s3 profile= blockstore s3 region=us east 1 blockstore s3 server side encryption= blockstore s3 server side encryption kms key id= blockstore s3 skip verify certificate test only=false blockstore s3 web identity session duration=0s blockstore s3 web identity session expiry window=5m0s blockstore type=s3 committed block storage prefix= lakefs committed local cache dir= ~ lakefs data cache committed local cache max uploaders per writer=10 committed local cache metarange proportion=0 1 committed local cache range proportion=0 9 committed local cache size bytes=1073741824 committed permanent max range size bytes=20971520 committed permanent min range size bytes=0 committed permanent range raggedness entries=50000 committed sstable memory cache size bytes=400000000 database drop tables=false database dynamodb aws access key id= database dynamodb aws profile= database dynamodb aws region= database dynamodb aws secret access key= database dynamodb endpoint= database dynamodb health check interval=0s database dynamodb scan limit=1024 database dynamodb table name=kvstore database local enable logging=false database local path= ~ lakefs metadata database local prefetch size=256 database local sync writes=true database postgres connection max lifetime=5m0s database postgres connection string= database postgres max idle connections=25 database postgres max open connections=25 database postgres metrics=false database postgres scan page size=0 database type=local diff delta plugin= email subscription enabled=true fields file= etc lakefs config yaml file= cmd root go 159 gateways s3 domain name= s3 local lakefs io gateways s3 fallback url= gateways s3 region=us east 1 graveler background rate limit=0 graveler batch dbio transaction markers=false graveler commit cache expiry=10m0s graveler commit cache jitter=2s graveler commit cache size=50000 graveler ensure readable root namespace=true graveler repository cache expiry=5s graveler repository cache jitter=2s graveler repository cache size=1000 installation access key id= installation fixed id= installation secret access key= installation user name= listen address= 0 0 0 0 8000 logging audit log level=DEBUG logging file max size mb=102400 logging files keep=100 logging format=text logging level=TRACE logging output= logging trace request headers=false phase=startup plugins default path= ~ lakefs plugins plugins properties= map security audit check interval=24h0m0s security audit check url= security check latest version=true security check latest version cache=1h0m0s stats address= stats enabled=false stats extended=false stats flush interval=30s stats flush size=100 tls cert file= tls enabled=false tls key file= ugc prepare interval=1m0s ugc prepare max file size=20971520 ui enabled=true ui snippets= time= 2023 10 20T21 08 50Z level=info msg= lakeFS run func=cmd lakefs cmd glob func8 file= cmd run go 91 version=0 113 0 time= 2023 10 20T21 08 50Z level=info msg= initialized Auth service func=pkg auth NewAuthService file= build pkg auth service go 187 service=auth service time= 2023 10 20T21 08 50Z level=debug msg= failed to collect account metadata func=pkg stats NewMetadata file= build pkg stats metadata go 34 error= not found ```

Itai Admi · Answer

And then does it exit

gang ye · Answer

pod keeps restarting with read probe and liveness probe failure

Itai Admi · Answer

Gotcha I don t think it has to do with the block adapter

Itai Admi · Answer

Do you have logs from the failing probes

gang ye · Answer

The above log is from the failing pod Probe it shows ```Liveness probe failed Get <http 172 16 95 57 8000 health> dial tcp 172 16 95 57 8000 connect connection refused``` I guess it s caused by server is not running so connection won t work

Itai Admi · Answer

What version are you using I ll try to reproduce it

gang ye · Answer

helm version 0 13 3 lakefs image version 0 113 0

gang ye · Answer

I have enabled TRACE log but there is no very helpful clue from the log The reason I think it s related to loading AWS credential especially for service account case is if I set the blockstore to local or set the aws credential access key secret key manually in lakefsConfig it works

Itai Admi · Answer

OK that s weird I m also unable to reproduce at the moment

Itai Admi · Answer

When the AWS creds are not set properly lakeFS normally 1 Exists on startup with the blockstore error 2 Fail to create a repo perform any data operation on existing repos

gang ye · Answer

any config we can set to enable AWS debug log looks like `logging level` in lakeConfig only enables lakefs server debug log

Itai Admi · Answer

Checking

Itai Admi · Answer

Yes can you try this lakeFS config ```lakefsConfig | logging level TRACE stats enabled false database type local blockstore type s3 s3 client log retries true client log request true``` Also please share ```printenv | grep LAKEFS```

gang ye · Answer

Sure one sec

gang ye · Answer

```~ $ printenv | grep LAKEFS LAKEFSINGRESS PORT= LAKEFS SERVICE HOST=10 100 145 100 LAKEFSINGRESS SERVICE PORT=80 LAKEFSINGRESS PORT 80 TCP ADDR=10 100 146 226 LAKEFS SERVICE PORT=80 LAKEFS PORT= LAKEFSINGRESS PORT 80 TCP PORT=80 LAKEFSINGRESS PORT 80 TCP PROTO=tcp LAKEFS PORT 80 TCP ADDR=10 100 145 100 LAKEFS PORT 80 TCP PORT=80 LAKEFS PORT 80 TCP PROTO=tcp LAKEFSINGRESS PORT 80 TCP= LAKEFS PORT 80 TCP= LAKEFSINGRESS SERVICE PORT HTTP=80 LAKEFS AUTH ENCRYPT SECRET KEY=123 LAKEFS SERVICE PORT HTTP=80 LAKEFSINGRESS SERVICE HOST=10 100 146 226```

gang ye · Answer

pod log message is same as before ```time= 2023 10 20T21 34 36Z level=info msg= Configuration file func= file= build cmd lakefs cmd root go 109 fields file= etc lakefs config yaml file= build cmd lakefs cmd root go 109 phase=startup time= 2023 10 20T21 34 36Z level=info msg= Config loaded func=cmd lakefs cmd initConfig file= cmd root go 151 fields file= etc lakefs config yaml file= cmd root go 151 phase=startup time= 2023 10 20T21 34 36Z level=info msg=Config func=cmd lakefs cmd initConfig file= cmd root go 159 actions enabled=true actions lua net http enabled=false auth api endpoint= auth api supports invites=false auth api token= auth cache enabled=true auth cache jitter=3s auth cache size=1024 auth cache ttl=20s auth cookie auth verification auth source= auth cookie auth verification default initial groups= auth cookie auth verification external user id claim name= auth cookie auth verification friendly name claim name= auth cookie auth verification initial groups claim name= auth cookie auth verification validate id token claims= map auth encrypt secret key= auth login duration=168h0m0s auth logout redirect url= auth login auth oidc default initial groups= auth oidc friendly name claim name= auth oidc initial groups claim name= auth oidc validate id token claims= map auth remote authenticator default user group=Viewers auth remote authenticator enabled=false auth remote authenticator endpoint= auth remote authenticator request timeout=10s auth ui config login cookie names= internal auth session auth ui config login failed message= The credentials don t match auth ui config login url= auth ui config logout url= auth ui config rbac=simplified blockstore azure auth method= blockstore azure disable pre signed=false blockstore azure disable pre signed ui=true blockstore azure pre signed expiry=15m0s blockstore azure storage access key= blockstore azure storage account= blockstore azure test endpoint url= blockstore azure try timeout=10m0s blockstore gs credentials file= blockstore gs credentials json= blockstore gs disable pre signed=false blockstore gs disable pre signed ui=true blockstore gs pre signed expiry=15m0s blockstore gs s3 endpoint= blockstore local allowed external prefixes= blockstore local import enabled=false blockstore local import hidden=false blockstore local path= ~ lakefs data block blockstore s3 client log request=true blockstore s3 client log retries=true blockstore s3 credentials file= blockstore s3 disable pre signed=false blockstore s3 disable pre signed ui=true blockstore s3 discover bucket region=true blockstore s3 endpoint= blockstore s3 force path style=false blockstore s3 max retries=5 blockstore s3 pre signed expiry=15m0s blockstore s3 profile= blockstore s3 region=us east 1 blockstore s3 server side encryption= blockstore s3 server side encryption kms key id= blockstore s3 skip verify certificate test only=false blockstore s3 web identity session duration=0s blockstore s3 web identity session expiry window=5m0s blockstore type=s3 committed block storage prefix= lakefs committed local cache dir= ~ lakefs data cache committed local cache max uploaders per writer=10 committed local cache metarange proportion=0 1 committed local cache range proportion=0 9 committed local cache size bytes=1073741824 committed permanent max range size bytes=20971520 committed permanent min range size bytes=0 committed permanent range raggedness entries=50000 committed sstable memory cache size bytes=400000000 database drop tables=false database dynamodb aws access key id= database dynamodb aws profile= database dynamodb aws region= database dynamodb aws secret access key= database dynamodb endpoint= database dynamodb health check interval=0s database dynamodb scan limit=1024 database dynamodb table name=kvstore database local enable logging=false database local path= ~ lakefs metadata database local prefetch size=256 database local sync writes=true database postgres connection max lifetime=5m0s database postgres connection string= database postgres max idle connections=25 database postgres max open connections=25 database postgres metrics=false database postgres scan page size=0 database type=local diff delta plugin= email subscription enabled=true fields file= etc lakefs config yaml file= cmd root go 159 gateways s3 domain name= gateways s3 fallback url= gateways s3 region=us east 1 graveler background rate limit=0 graveler batch dbio transaction markers=false graveler commit cache expiry=10m0s graveler commit cache jitter=2s graveler commit cache size=50000 graveler ensure readable root namespace=true graveler repository cache expiry=5s graveler repository cache jitter=2s graveler repository cache size=1000 installation access key id= installation fixed id= installation secret access key= installation user name= listen address= 0 0 0 0 8000 logging audit log level=DEBUG logging file max size mb=102400 logging files keep=100 logging format=text logging level=TRACE logging output= logging trace request headers=false phase=startup plugins default path= ~ lakefs plugins plugins properties= map security audit check interval=24h0m0s security audit check url= security check latest version=true security check latest version cache=1h0m0s stats address= stats enabled=false stats extended=false stats flush interval=30s stats flush size=100 tls cert file= tls enabled=false tls key file= ugc prepare interval=1m0s ugc prepare max file size=20971520 ui enabled=true ui snippets= time= 2023 10 20T21 34 36Z level=info msg= lakeFS run func=cmd lakefs cmd glob func8 file= cmd run go 91 version=0 113 0 time= 2023 10 20T21 34 36Z level=info msg= initialized Auth service func=pkg auth NewAuthService file= build pkg auth service go 187 service=auth service time= 2023 10 20T21 34 36Z level=debug msg= failed to collect account metadata func=pkg stats NewMetadata file= build pkg stats metadata go 34 error= not found ```

Itai Admi · Answer

Not sure it s related but still worth a shot How many instances of lakeFS pods are you using

gang ye · Answer

only one instance

Itai Admi · Answer

Oh what are the values you re using for the probes Did you uncomment the `livenessProbe` amp `readinessProbe` section in the `values yaml` file

gang ye · Answer

I didn t uncomment the livenessProbe and readinessProbe part

Itai Admi · Answer

Can you try that perhaps it takes a bit longer for lakeFS to spinup with the service account

gang ye · Answer

uncomment ```extraEnvVars Override K8S defaults for readinessProbe readinessProbe failureThreshold 10 periodSeconds 5 successThreshold 4 timeoutSeconds 1 Override K8S defaults for livenessProbe livenessProbe failureThreshold 20 periodSeconds 5 successThreshold 4 timeoutSeconds 1 initialDelaySeconds 5``` but there is format issue

gang ye · Answer

if it takes time to spinup service account then when pod restarts I think it should work But what i saw is it never succeeds

Itai Admi · Answer

It s the initialization of the S3 blockstore with service account that takes longer and that happens with every pod restart At least that s my theory

Itai Admi · Answer

I think you need to ident it left no whitespace before `readinessProbe` amp `livenessProbe`

gang ye · Answer

this is the default value for the probe I will update it manually in k8s deployment ``` livenessProbe httpGet path health port http scheme HTTP timeoutSeconds 1 periodSeconds 10 successThreshold 1 failureThreshold 3 readinessProbe httpGet path health port http scheme HTTP timeoutSeconds 1 periodSeconds 10 successThreshold 1 failureThreshold 3```

gang ye · Answer

update to below values and restarting deployment ``` livenessProbe httpGet path health port http scheme HTTP timeoutSeconds 5 periodSeconds 10 successThreshold 1 failureThreshold 10 readinessProbe httpGet path health port http scheme HTTP timeoutSeconds 5 periodSeconds 10 successThreshold 1 failureThreshold 10```

gang ye · Answer

same error ```Liveness probe failed Get <http 172 16 150 9 8000 health> dial tcp 172 16 150 9 8000 connect connection refused```

Itai Admi · Answer

I m out of ideas at the moment I ll consult my colleagues and provide an answer early next week I understand you have a workaround for now right the static creds

gang ye · Answer

we don t have a long existing s3 creds security rule only a temp one which will expire every few hours

Itai Admi · Answer

Got it so it s indeed a blocker

gang ye · Answer

yeah thanks very much for help

Isan Rivkin · Answer

Hey < gang ye> Regarding the the credentials you don t need to use static credentials Depends on how you perform auth we internally run on for example and that means that by adding an annotation to the serviceAccount that is attached to the lakeFS Deployment we give it all the permission it needs It s basically based on IAM AssumeRole and there s no static credentials involved You can even configure this in the Helm chart itself including the SA creation for example that s how that part would look in `values yaml` ```serviceAccount name my svc acc extraManifests apiVersion v1 kind ServiceAccount metadata name Values serviceAccount name annotations eks amazonaws com role arn arn aws iam ACCOUNT ID role my pod role ```

gang ye · Answer

Hi < Isan Rivkin> yes we used service account with right annotation setting The issue is once using service account lakefs server cannot start as expected Logs are attached earlier

Isan Rivkin · Answer

So if I understood correctly the issue is that the liveness probe causes restarts due to some unknown reason I don t see anything helpful at the logs is it correct If so can you maybe try and override the container command to sleep 1d of lakeFS through the chart and give some infinite amount for the readiness Then once its up exec into the container and try to to run lakefs Then outside of the container try to reach it with curl or something like that Also do you have an HTTP Proxy in your environment Is the blockstore S3 or some compatible interface

gang ye · Answer

I think the liveness probe and read probe themselves are not the root cause of the problem The root cause is lakefs server cannot start when using service account log message doesn t provide helpful clue and the blockstore i m using is S3 And I have used the same service account on other pod to access s3 which works as expected

Isan Rivkin · Answer

Could be let s explore that option Can you please make sure that your namespace and service account information is correct The the SA needs to be in the same namespace and the N needs to exist before the SA is created If you configured it to use a default SA which I don t think is your case but just to make sure then that has to exist as well and the SA should be created after the namespace Finally if none of that helps can you please attach the events of the resources I m pretty sure we will see the answer there ```k describe deploy k describe pod try to catch pod events before it restarts k describe replicaset k describe sa k describe svc ```

gang ye · Answer

Hi < Isan Rivkin> Thanks for following up on the issue The the SA needs to be in the same namespace and the N needs to exist before the SA is created yes SA exists in same namespace If you configured it to use a default SA which I don t think is your case but just to make sure then that has to exist as well and the SA should be created after the namespace We don t use default SA describe deploy ```kubectl describe deploy lakefstest n data experimentation Name lakefstest Namespace data experimentation CreationTimestamp Mon 23 Oct 2023 17 33 35 0700 Labels app=lakefs Annotations 1 lakefstest data experimentation Selector app=lakefs Replicas 1 desired | 1 updated | 1 total | 0 available | 1 unavailable StrategyType RollingUpdate MinReadySeconds 0 RollingUpdateStrategy 25 max unavailable 25 max surge Pod Template Labels app=lakefs Annotations checksum config 5b7e985282116d067aa462b11debaeb3fc43e4fcdd55194645756e057e9bcc89 Service Account data experimentation sa Containers lakefs Image Port 8000 TCP Host Port 0 TCP Args run config etc lakefs config yaml Liveness http get http http health delay=0s timeout=1s period=10s success=1 failure=3 Readiness http get http http health delay=0s timeout=1s period=10s success=1 failure=3 Environment LAKEFS AUTH ENCRYPT SECRET KEY lt set to the key auth encrypt secret key in secret lakefstest gt Optional false Mounts etc lakefs from config volume rw Volumes config volume Type ConfigMap a volume populated by a ConfigMap Name lakefstest Optional false Conditions Type Status Reason Available False MinimumReplicasUnavailable Progressing True ReplicaSetUpdated OldReplicaSets lt none gt NewReplicaSet lakefstest 5c9c5b454b 1 1 replicas created Events Type Reason Age From Message Normal ScalingReplicaSet 28s deployment controller Scaled up replica set lakefstest 5c9c5b454b to 1``` Describe pod ```kubectl describe pod $POD NAME n data experimentation Name lakefstest 5c9c5b454b hd7bj Namespace data experimentation Priority 0 Service Account data experimentation sa Node ip 172 16 162 137 us west 2 compute internal 172 16 162 137 Start Time Mon 23 Oct 2023 17 33 36 0700 Labels app=lakefs pod template hash=5c9c5b454b Annotations checksum config 5b7e985282116d067aa462b11debaeb3fc43e4fcdd55194645756e057e9bcc89 00 fully open Status Running IP 172 16 134 189 IPs IP 172 16 134 189 Controlled By ReplicaSet lakefstest 5c9c5b454b Containers lakefs Container ID 1 Image Image ID Port 8000 TCP Host Port 0 TCP Args run config etc lakefs config yaml State Running Started Mon 23 Oct 2023 17 34 37 0700 Last State Terminated Reason Error Exit Code 137 Started Mon 23 Oct 2023 17 33 36 0700 Finished Mon 23 Oct 2023 17 34 36 0700 Ready False Restart Count 1 Liveness http get http http health delay=0s timeout=1s period=10s success=1 failure=3 Readiness http get http http health delay=0s timeout=1s period=10s success=1 failure=3 Environment LAKEFS AUTH ENCRYPT SECRET KEY lt set to the key auth encrypt secret key in secret lakefstest gt Optional false AWS STS REGIONAL ENDPOINTS regional AWS DEFAULT REGION us west 2 AWS REGION us west 2 AWS ROLE ARN arn aws iam xxx role aiml data experimentation AWS WEB IDENTITY TOKEN FILE var run secrets eks amazonaws com serviceaccount token Mounts etc lakefs from config volume rw var run secrets eks amazonaws com serviceaccount from aws iam token ro var run secrets kubernetes io serviceaccount from kube api access 49t2q ro Conditions Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes aws iam token Type Projected a volume that contains injected data from multiple sources TokenExpirationSeconds 86400 config volume Type ConfigMap a volume populated by a ConfigMap Name lakefstest Optional false kube api access 49t2q Type Projected a volume that contains injected data from multiple sources TokenExpirationSeconds 3607 ConfigMapName kube root ca crt ConfigMapOptional lt nil gt DownwardAPI true QoS Class BestEffort Node Selectors lt none gt Tolerations op=Exists for 300s op=Exists for 300s Events Type Reason Age From Message Normal Scheduled 84s default scheduler Successfully assigned data experimentation lakefstest 5c9c5b454b hd7bj to ip 172 16 162 137 us west 2 compute internal Normal Killing 54s kubelet Container lakefs failed liveness probe will be restarted Normal Pulled 23s x2 over 84s kubelet Container image already present on machine Normal Created 23s x2 over 84s kubelet Created container lakefs Normal Started 23s x2 over 84s kubelet Started container lakefs Warning Unhealthy 4s x13 over 83s kubelet Readiness probe failed Get dial tcp 172 16 134 189 8000 connect connection refused Warning Unhealthy 4s x5 over 74s kubelet Liveness probe failed Get dial tcp 172 16 134 189 8000 connect connection refused```

gang ye · Answer

describe svc ```kubectl describe svc lakefstest n data experimentation Name lakefstest Namespace data experimentation Labels app=lakefs Annotations lakefstest data experimentation Selector Type ClusterIP IP Family Policy SingleStack IP Families IPv4 IP 10 100 178 193 IPs 10 100 178 193 Port http 80 TCP TargetPort http TCP Endpoints Session Affinity None Events lt none gt ``` describe sa ```kubectl describe sa data experimentation sa n data experimentation Name data experimentation sa Namespace data experimentation Labels lt none gt Annotations arn aws iam xxx role aiml data experimentation Image pull secrets lt none gt Mountable secrets data experimentation sa token 4m9d5 Tokens data experimentation sa token 4m9d5 Events lt none gt ```

gang ye · Answer

if you need the helm config file ```image repository repository tag 0 113 0 amd64 pullPolicy IfNotPresent Keys used for existingSecret secrets authEncryptSecretKey 123 lakefsConfig | logging level TRACE stats enabled false database type local blockstore type s3 s3 client log retries true client log request true serviceAccount name data experimentation sa```

gang ye · Answer

And in pod log message there is no message like below indicating blockstore is set up successfully ```time= 2023 10 21T00 15 18Z level=info msg= initialize blockstore adapter func=pkg block factory BuildBlockAdapter file= build pkg block factory build go 32 type=s3 time= 2023 10 21T00 15 18Z level=info msg= initialized blockstore adapter func=pkg block factory buildS3Adapter file= build pkg block factory build go 111 type=s3```

Isan Rivkin · Answer

Hey < gang ye> thanks for getting back to me and for the patience with the delay in response it s due to time zone difference After Inspecting the additional info you have provided i I have noticed few suspects but nothing concrete 1 First I must say that if you prefer we can do a short meeting to troubleshoot see DM computer 2 I have released from a a lakeFS version 1 0 0 with 1 change very extensive logging of all the steps in the setup process That way we can see where it get s stuck hammer a Can you please update the Helm Chart values with the docker image `` and tag `1 0 0 vvv1` attached full `values yaml` below b All the logs I added contain the field `trace flow=true` please run the new chart and attach the logs If possible re run twice to make sure we don t crush on random places in the code based on the last log message we will see 3 Regarding the issue itself male detective the latest logs you sent me I think that K8S cannot locate runtime dependencies i e the var run secrets kubernetes io or service account files are missing a What K8S Version is the cluster b Are you using any special plugins for authentication authz between pods i e calico c This might occur when some containers inside the pod attempt to interact with an API without the default access token d I suspect it because the ServiceAccount you attached has values set in `Mounted Secrets` and that s an old way of accessing SA tokens e If that s the case it can fix this error by allowing all new mount creations to adhere to the default access level throughout the pod space Ensure that new pods using custom tokens comply with this access level to prevent continuous startup failures This can be done with setting ` Values podSecurityContext` and ` Values securityContext` please refer to this as a nice reference to the values and maybe check other charts you have in the cluster ```image repository tag 1 0 0 vvv1 pullPolicy IfNotPresent Keys used for existingSecret secrets authEncryptSecretKey 123 lakefsConfig | logging level DEBUG stats enabled false database type local blockstore type s3 s3 client log retries true client log request true 90 seconds grace to start maybe something will pop up in the logs livenessProbe initialDelaySeconds 90 serviceAccount name data experimentation lakefs sa this will create a new service account extraManifests apiVersion v1 kind ServiceAccount metadata name data experimentation lakefs sa annotations set the correct ARN arn aws iam xxx role aiml data experimentation ```

gang ye · Answer

Hi < Isan Rivkin> Thanks for the detail I will redeploy with image 1 0 0 vvv1 today and then get back to you to see if we can find any clue Regarding the issue we don t use any special authn authz For the service account token I once logged into the pod and I can print it out var run secrets eks amazonaws com serviceaccount token successfully K8s version ```kubectl version short Flag short has been deprecated and will be removed in the future The short output will become the default Client Version v1 25 9 Kustomize Version v4 5 7 Server Version v1 23 17 eks 2d98532```

gang ye · Answer

Hi < Isan Rivkin> One quick question does the new image 1 0 0 vvv1 support amd64

Isan Rivkin · Answer

I built it specifically for amd64 yes why

gang ye · Answer

Got it Then it should work Our k8s arch is amd64

gang ye · Answer

Hi < Isan Rivkin> Below is the log message running on 1 0 0 vvv1 ```time= 2023 10 24T18 04 07Z level=info msg= starting build of block adapter func=pkg block factory BuildBlockAdapter file= build pkg block factory build go 31 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= blockstore type s3 func= pkg logging logrusEntryWrapper Infof file= build pkg logging logger go 272 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= initialize blockstore adapter func=pkg block factory BuildBlockAdapter file= build pkg block factory build go 36 type=s3 time= 2023 10 24T18 04 07Z level=info msg= initialized blockstore adapter func=pkg block factory buildS3Adapter file= build pkg block factory build go 115 type=s3 time= 2023 10 24T18 04 07Z level=info msg= finished block adapter build starting runtime collector func=cmd lakefs cmd glob func8 file= cmd run go 164 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= post SetRuntimeCollector func=cmd lakefs cmd glob func8 file= cmd run go 166 trace flow=true time= 2023 10 24T18 04 07Z level=trace msg= dummy sender received metadata func= pkg stats dummySender UpdateMetadata file= build pkg stats sender go 147 metadata= InstallationID d42db8a3 e6ec 4fdd b5d1 fa06564a2e8b Entries Name is docker Value true Name instrumentation Value Run Name lakefs version Value dev Name lakefs kv type Value local Name golang version Value go1 20 6 Name os Value linux Name architecture Value amd64 Name is k8s Value true Name installation id Value d42db8a3 e6ec 4fdd b5d1 fa06564a2e8b Name blockstore type Value s3 service=stats collector time= 2023 10 24T18 04 07Z level=info msg= post CollectMetadata initiating catalog func=cmd lakefs cmd glob func8 file= cmd run go 169 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= head build block adapter trace flow true func= pkg logging logrusEntryWrapper Infof file= build pkg logging logger go 272 time= 2023 10 24T18 04 07Z level=info msg= starting build of block adapter func=pkg block factory BuildBlockAdapter file= build pkg block factory build go 31 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= blockstore type s3 func= pkg logging logrusEntryWrapper Infof file= build pkg logging logger go 272 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= initialize blockstore adapter func=pkg block factory BuildBlockAdapter file= build pkg block factory build go 36 type=s3 time= 2023 10 24T18 04 07Z level=info msg= initialized blockstore adapter func=pkg block factory buildS3Adapter file= build pkg block factory build go 115 type=s3 time= 2023 10 24T18 04 07Z level=info msg= Post Catalog Initialization func=cmd lakefs cmd glob func8 file= cmd run go 178 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= Pre scheduler func=cmd lakefs cmd glob func8 file= cmd run go 180 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= Pre scheduler deleteScheduler StartAsync func=cmd lakefs cmd glob func8 file= cmd run go 186 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= Pre new actions service func=cmd lakefs cmd glob func8 file= cmd run go 202 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= Post new actions service func=cmd lakefs cmd glob func8 file= cmd run go 212 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= pre middlewareAuthenticator func=cmd lakefs cmd glob func8 file= cmd run go 217 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= post middlewareAuthenticator func=cmd lakefs cmd glob func8 file= cmd run go 221 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= pre NewDefaultAuditChecker func=cmd lakefs cmd glob func8 file= cmd run go 231 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= post NewDefaultAuditChecker func=cmd lakefs cmd glob func8 file= cmd run go 235 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= pre checkRepos func=cmd lakefs cmd glob func8 file= cmd run go 245 trace flow=true time= 2023 10 24T18 04 07Z level=debug msg= lakeFS isn t initialized skipping mismatched adapter checks func=cmd lakefs cmd checkRepos file= cmd run go 387 time= 2023 10 24T18 04 07Z level=info msg= post checkRepos func=cmd lakefs cmd glob func8 file= cmd run go 247 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= pre updating SetHealthHandlerInfo func=cmd lakefs cmd glob func8 file= cmd run go 249 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= post updating SetHealthHandlerInfo func=cmd lakefs cmd glob func8 file= cmd run go 252 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= pre init api Serve func=cmd lakefs cmd glob func8 file= cmd run go 255 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= initialize OpenAPI server func=pkg api Serve file= build pkg api serve go 38 service=api gateway time= 2023 10 24T18 04 07Z level=info msg= post init api Serve func=cmd lakefs cmd glob func8 file= cmd run go 275 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= pre SSO auth middlewares func=cmd lakefs cmd glob func8 file= cmd run go 285 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= post SSO auth middlewares func=cmd lakefs cmd glob func8 file= cmd run go 300 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= initialized S3 Gateway handler func=pkg gateway NewHandler file= build pkg gateway handler go 124 s3 bare domain= s3 local lakefs io s3 region=us east 1 time= 2023 10 24T18 04 07Z level=info msg= pre apiAuthenticator func=cmd lakefs cmd glob func8 file= cmd run go 314 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= post apiAuthenticator func=cmd lakefs cmd glob func8 file= cmd run go 316 trace flow=true time= 2023 10 24T18 04 07Z level=info msg= starting HTTP server func=cmd lakefs cmd glob func8 file= cmd run go 322 listen address= 0 0 0 0 8000 ``` It looks like the server starts successfully I will verify the s3 access

gang ye · Answer

I verified it works now It looks like once bumping up version from 0 113 0 to 1 0 0 solve the issue

Isan Rivkin · Answer

Yyyyeess raised hands