Hi, im trying to run the lakefs server but i keep getting this error when i try to login (i removed my email for privacy) Any thoughts?

hi Natasha. How are you running lakeFS (e.g. local binary, Docker, etc)? And which version?

im running with docker via codespaces using the latest version

Hey @Natasha Taylor, I apologize for the inconvenience. Since some bugs around the setup page have recently been fixed, can you make sure that the latest lakeFS image is pulled?

hi @Yoni Augarten when i run the code im running this which im assuming is pulling the latest lakeFS image

Hi @Natasha Taylor, try replacing the docker image tag to a specific version (i.e replace `treeverse/lakefs:latest`` with

treeverse/lakefs:0.95.0

)

(not to take this thread off topic, but I wonder if we should update https://docs.lakefs.io/deploy/azure.html#storage-account-access to also include

--pull always

in the

docker run

like we have for the quickstart)

Since this one is a production deployment guide, I'm not sure. On the other hand, using "latest" in a production guide is also not the best idea probably.

Hi so it's still giving me the same error after changing to treeverse/lakefs:0.95.0 In the terminal window i get this - would it be anything to do with this or is this a different issue?

level=warning msg="Failed to get Azure subscription ID from instance metadataGet \

@Natasha Taylor this warn message probably is not the root cause of the failure. Can you list the complete log (if possible) or errors you see? Also what is the UI behaviour - refresh the browser are you still in the setup page?

level=warning msg="Failed to get Azure subscription ID from instance metadataGet \"XXX": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" func="pkg/cloud/azure.(*MetadataProvider).GetMetadata" file="build/pkg/cloud/azure/metadata.go:45" error="Get \"XXX": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Yes if i refresh i'm still in the setup page

1. no other messages in the output? 2. from the warning it looks like it fails to communicate with your azure account did the previous setup use azure blob storage?

auth.ui_config.login_failed_message="The credentials don't match."

There is this in the output which could be related! Although i've triple checked my credentials and they are correct

will it be possible to upload the full output of the server - without any sensitive information?

Hi Natasha and Barak, Have you been able to figure this out? I'm trying to run LakeFS in an Azure Container Instance and connecting to Azure Datalake Gen2 blob storage and think I'm having a similar issue... I'm getting the same Metadata timeout and related warning about not being able to fetch the Azure subscription ID on start-up. Then, whenever I try to create a repo in the LakeFS web UI, I get below nil pointer dereference error. Thanks in advance for any assistance or info provided! 2023/08/06 040008 http: panic serving <MY-IP>64413 runtime error: invalid memory address or nil pointer dereference goroutine 1285 [running]: net/http.(*conn).serve.func1() /usr/local/go/src/net/http/server.go:1850 +0xbf panic({0x1d48820, 0x63f5ac0}) /usr/local/go/src/runtime/panic.go:890 +0x262 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/base.InnerClient[...](0x100?) /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.6.1/internal/base/clients.go:21 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/service.(*Client).generated(...) /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.6.1/service/client.go:111 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/service.(*Client).NewContainerClient(0xc0004a53e0?, {0xc0071e8132?, 0x0?}) /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.6.1/service/client.go:126 +0x3f github.com/treeverse/lakefs/pkg/block/azure.(*ClientCache).NewContainerClient.func1() /build/pkg/block/azure/client_cache.go:56 +0x90 github.com/puzpuzpuz/xsync.(*MapOf[...]).doStore(0xc0003c92d0, {0xc0076d1890, 0x15}, 0xc007e3e2e8, 0x1) /go/pkg/mod/github.com/puzpuzpuz/xsync@v1.5.2/mapof.go:238 +0x63e github.com/puzpuzpuz/xsync.(*MapOf[...]).LoadOrCompute(0x5?, {0xc0076d1890?, 0x2?}, 0x2?) /go/pkg/mod/github.com/puzpuzpuz/xsync@v1.5.2/mapof.go:161 +0x28 github.com/treeverse/lakefs/pkg/block/azure.(*ClientCache).NewContainerClient(0xc0004a53e0, {0xc0071e8108, 0x14}, {0xc0071e8132, 0x0}) /build/pkg/block/azure/client_cache.go:50 +0x156 github.com/treeverse/lakefs/pkg/block/azure.(*Adapter).Download(0xc0007fadb0, {0x5565290, 0xc0077e6540}, {{0xc0071e8100, 0x33}, {0x20af1b4, 0x5}, 0x1}, 0x0, 0x0) /build/pkg/block/azure/adapter.go:285 +0x169 github.com/treeverse/lakefs/pkg/block/azure.(*Adapter).Get(0x14?, {0x5565290, 0xc0077e6540}, {{0xc0071e8100, 0x33}, {0x20af1b4, 0x5}, 0x1}, 0xc0075d4600?) /build/pkg/block/azure/adapter.go:195 +0x145 github.com/treeverse/lakefs/pkg/api.(*Controller).ensureStorageNamespace(0xc00721b500, {0x5565290, 0xc0077e6540}, {0xc0071e8100, 0x33}) /build/pkg/api/controller.go:1620 +0xcf github.com/treeverse/lakefs/pkg/api.(*Controller).CreateRepository(0xc00721b500, {0x55632d0, 0xc0073d2090}, 0xc0075d4600, {0xc007d48140, {0xc0008521a0, 0xc}, 0xc00085213c, {0xc0071e8100, 0x33}}, ...) /build/pkg/api/controller.go:1542 +0x572 github.com/treeverse/lakefs/pkg/api.(*ServerInterfaceWrapper).CreateRepository.func1({0x55632d0?, 0xc0073d2090?}, 0x20a0001?) /build/pkg/api/lakefs.gen.go:21059 +0x65 github.com/treeverse/lakefs/pkg/api.(*ServerInterfaceWrapper).CreateRepository(0xc00753a810, {0x55632d0, 0xc0073d2090}, 0xc0075d4500) /build/pkg/api/lakefs.gen.go:21066 +0x5b0 net/http.HandlerFunc.ServeHTTP(0xc007b41970?, {0x55632d0?, 0xc0073d2090?}, 0xc0077e6330?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/treeverse/lakefs/pkg/api.MetricsMiddleware.func1.1({0x7f0be1471348?, 0xc0077e6330}, 0x5563300?) /build/pkg/api/metrics_middleware.go:27 +0xf3 net/http.HandlerFunc.ServeHTTP(0x5563300?, {0x7f0be1471348?, 0xc0077e6330?}, 0xc0075d4500?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x5563300?, 0xc007d7c620?}, 0xc0075d4500) /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:147 +0xc5 net/http.HandlerFunc.ServeHTTP(0x5565290?, {0x5563300?, 0xc007d7c620?}, 0x5540fe0?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/treeverse/lakefs/pkg/api.AuthMiddleware.func1.1({0x5563300, 0xc007d7c620}, 0xc00747e800) /build/pkg/api/auth_middleware.go:93 +0x4b0 net/http.HandlerFunc.ServeHTTP(0x1f08860?, {0x5563300?, 0xc007d7c620?}, 0xc?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/treeverse/lakefs/pkg/httputil.DefaultLoggingMiddleware.func1.1({0x5563c00?, 0xc0073f4000}, 0xffffffffffffff01?) /build/pkg/httputil/logging.go:87 +0x645 net/http.HandlerFunc.ServeHTTP(0xc00747e500?, {0x5563c00?, 0xc0073f4000?}, 0xc0074f8c50?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/treeverse/lakefs/pkg/api.OapiRequestValidatorWithOptions.func1.1({0x5563c00, 0xc0073f4000}, 0xc00747e400) /build/pkg/api/serve.go:168 +0x2e3 net/http.HandlerFunc.ServeHTTP(0xc007b21f80?, {0x5563c00?, 0xc0073f4000?}, 0xc0080e8a38?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP(0x1d20d00?, {0x5563c00?, 0xc0073f4000?}, 0xc007170005?) /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.0/chain.go:31 +0x2c github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc00011d080, {0x5563c00, 0xc0073f4000}, 0xc00747e400) /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.0/mux.go:436 +0x1f9 net/http.HandlerFunc.ServeHTTP(0x55651e8?, {0x5563c00?, 0xc0073f4000?}, 0x63f54b0?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc00011d080, {0x5563c00, 0xc0073f4000}, 0xc007495b00) /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.0/mux.go:87 +0x32a github.com/treeverse/lakefs/cmd/lakefs/cmd.glob..func8.3({0x5563c00, 0xc0073f4000}, 0xc0077d86f0?) /build/cmd/lakefs/cmd/run.go:325 +0x102 net/http.HandlerFunc.ServeHTTP(0x0?, {0x5563c00?, 0xc0073f4000?}, 0x866854?) /usr/local/go/src/net/http/server.go:2109 +0x2f net/http.serverHandler.ServeHTTP({0x555e340?}, {0x5563c00, 0xc0073f4000}, 0xc007495b00) /usr/local/go/src/net/http/server.go:2947 +0x30c net/http.(*conn).serve(0xc007c47b80, {0x5565290, 0xc008102ff0}) /usr/local/go/src/net/http/server.go:1991 +0x607 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:3102 +0x4db

Hey @Sergio Moreta, what’s the lakeFS version you’re using? I’ll try to reproduce it.

Hi @Itai Admi, Thanks for the quick response and for offering to try to reproduce it! I'm pulling the latest version from Docker hub. Please let me know if you need any additional details of my deployment (attempt 😉).

Sometime the image

repo/image:latest

gets cached and you don’t run the actual latest version. Can you confirm it’s

0.104.0

?

Hi @Itai Admi, Apologies for the delay. Indeed, I'm running version 0.104.0 Thanks again.

Hi @Sergio Moreta, Can you please share the lakeFS configurations used to run the server and the format of the blob storage URI you used to create your repository? (hiding sensitive information of course) And just to be sure, the container that you use for the repository creation is under the storage account that you provided in the configurations of lakeFS, correct?

Hi @Jonathan Rosenberg, Okay, let me try... For the LakeFS configuration, the only information I'm providing to the container instance is through these environment variables: LAKEFS_DATABASE_TYPE="postgres" LAKEFS_BLOCKSTORE_TYPE="azure" LAKEFS_BLOCKSTORE_AZURE_STORAGE_ACCOUNT="<myblobstorageaccountname>" LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING="host=<mydbname>.postgres.database.azure.com user=<myuser>@<myinstancename>.postgres.database.azure.com password=************" LAKEFS_AUTH_ENCRYPT_SECRET_KEY="<randomkey>" LAKEFS_BLOCKSTORE_AZURE_STORAGE_ACCESS_KEY="<myblobdatacontributorpasswd>" For the URI that I'm providing in the "Create New Repository" window (just before the error) is: https://<myblobstorageaccountname>.blob.core.windows.net/raw/ Lastly, yes, indeed, I've only setup a single storage account, hence both instances of the token <myblobstorageaccountname> refer to the same name. Thank you.

Thanks I’ll take a look

Hey Jonathan, Thank you for the tips. It indeed seems that I was confusing the two authentication methods and was hence using the wrong key. It looks like I'm one step further now; The web GUI now reports the more verbose message "failed to create repository: failed to access storage" and the nil pointer dereference error is no longer being triggered. I'll also try switching to identity based authentication, as you suggest, since this is now clearly an authentication issue against my storage account. B.t.w. I see that the latest image pulled is now 0.105.0 and noticed some changes in the GUI. Thanks again for the kind assistance.

Glad to be of help. If you’re still facing issues, please let us know 🙏