Hi team I tried to deploy lakefs using helm onto k8s cluster lakeFS #help

Hi team, I tried to deploy lakefs using helm onto ...

gang ye

10/18/2023, 6:32 AM

Hi team, I tried to deploy lakefs using helm onto k8s cluster. But the pod failed on live probe and read probe with error

*dial tcp 172.19..***:8000: connect: connection refused*

The last messages I found in the pod are

Copy code

time="2023-10-18T06:26:08Z" level=info msg="lakeFS run" func=cmd/lakefs/cmd.glob..func8 file="cmd/run.go:91" version=0.113.0
time="2023-10-18T06:26:08Z" level=info msg="initialized Auth service" func=pkg/auth.NewAuthService file="build/pkg/auth/service.go:187" service=auth_service
time="2023-10-18T06:26:08Z" level=info msg="initialize blockstore adapter" func=pkg/block/factory.BuildBlockAdapter file="build/pkg/block/factory/build.go:32" type=local
time="2023-10-18T06:26:08Z" level=info msg="initialized blockstore adapter" func=pkg/block/factory.buildLocalAdapter file="build/pkg/block/factory/build.go:79" path=/home/lakefs/lakefs/data/block type=local

I don’t find any message like

"Initial setup completed successfully"

and

"starting HTTP server"

in the log. Here is my helm cmd `helm install example-lakefs-amd64 lakefs/lakefs --version 0.13.3 -f lakefs-config-helm-amd64-local.yaml`and the config file

Copy code

image:
  repository: <http://docker.io/treeverse/lakefs|docker.io/treeverse/lakefs>
  pullPolicy: IfNotPresent

# Keys used for existingSecret
secrets:
  authEncryptSecretKey: "123"

lakefsConfig: |
  database:
    type: "local"
  blockstore:
    type: "local"

Any wrong setting? Thanks!

gang ye

10/18/2023, 6:53 AM

Or is there any debug mode we can enable to see where it gets stuck which causes the HTTP server not running as expected?

Jonathan Rosenberg

10/18/2023, 7:23 AM

Hi @gang ye Would you mind trying to run it without the

lakefsConfig

and

image

sections? These are the default values anyway…

gang ye

10/18/2023, 4:19 PM

Hi @Jonathan Rosenberg Ideally the image config should not impact. Our k8s cluster can only connect with internal docker lib. So we would need to set it.

Jonathan Rosenberg

10/18/2023, 4:53 PM

Hi again, I used your configurations without the quotes that surround “local” and it worked. Can you make sure that the lakeFS pod is actually running?

gang ye

10/18/2023, 7:54 PM

I tried, still same. No log message saying `starting HTTP server`after

Copy code

time="2023-10-18T19:50:47Z" level=info msg="initialized Auth service" func=pkg/auth.NewAuthService file="build/pkg/auth/service.go:187" service=auth_service
time="2023-10-18T19:50:47Z" level=info msg="initialize blockstore adapter" func=pkg/block/factory.BuildBlockAdapter file="build/pkg/block/factory/build.go:32" type=local
time="2023-10-18T19:50:47Z" level=info msg="initialized blockstore adapter" func=pkg/block/factory.buildLocalAdapter file="build/pkg/block/factory/build.go:79" path=/home/lakefs/lakefs/data/block type=local

and the read probe and write probe refuse connection

gang ye

10/18/2023, 8:08 PM

I enabled trace log, here is the log

Copy code

time="2023-10-18T20:09:03Z" level=info msg="lakeFS run" func=cmd/lakefs/cmd.glob..func8 file="cmd/run.go:91" version=0.113.0
time="2023-10-18T20:09:03Z" level=info msg="initialized Auth service" func=pkg/auth.NewAuthService file="build/pkg/auth/service.go:187" service=auth_service
time="2023-10-18T20:09:03Z" level=debug msg="failed to collect account metadata" func=pkg/stats.NewMetadata file="build/pkg/stats/metadata.go:34" error="not found"
time="2023-10-18T20:09:03Z" level=info msg="initialize blockstore adapter" func=pkg/block/factory.BuildBlockAdapter file="build/pkg/block/factory/build.go:32" type=local
time="2023-10-18T20:09:03Z" level=info msg="initialized blockstore adapter" func=pkg/block/factory.buildLocalAdapter file="build/pkg/block/factory/build.go:79" path=/home/lakefs/lakefs/data/block type=local
time="2023-10-18T20:09:03Z" level=trace msg="[DEBUG] POST <https://stats.lakefs.io/installation>" func="pkg/logging.(*logrusEntryWrapper).Tracef" file="build/pkg/logging/logger.go:264" service=stats_collector
time="2023-10-18T20:09:33Z" level=trace msg="[ERR] POST <https://stats.lakefs.io/installation> request failed: Post \"<https://stats.lakefs.io/installation>\": dial tcp 54.175.118.214:443: i/o timeout" func="pkg/logging.(*logrusEntryWrapper).Tracef" file="build/pkg/logging/logger.go:264" service=stats_collector
time="2023-10-18T20:09:33Z" level=trace msg="[DEBUG] POST <https://stats.lakefs.io/installation>: retrying in 1s (2 left)" func="pkg/logging.(*logrusEntryWrapper).Tracef" file="build/pkg/logging/logger.go:264" service=stats_collector

gang ye

10/18/2023, 8:16 PM

after I disabled stats, then it works. I guess it’s related to our internal K8s cluster cannot connect with lakefs stats.

gang ye

10/18/2023, 8:40 PM

Hi @Jonathan Rosenberg One last question, after running

Copy code

kubectl port-forward $POD_NAME 8000:8000 --namespace default

how to access the local port to create repo? Is this the right link http://localhost:8000/repositories?

Amit Kesarwani

10/18/2023, 8:46 PM

Try setup: http://127.0.0.1:8000/setup or http://localhost:8000/setup

👍 1

Iddo Avneri

10/19/2023, 6:48 PM

@gang ye - Is your environment now working correctly?

gang ye

10/19/2023, 9:11 PM

it’s working as expected. Thanks!

🙏 1

Open in Slack

Previous Next