https://lakefs.io/ logo
#help
Title
# help
n

Natasha Taylor

03/08/2023, 11:52 AM
Hi, im trying to run the lakefs server but i keep getting this error when i try to login (i removed my email for privacy) Any thoughts?
r

Robin Moffatt

03/08/2023, 11:56 AM
hi Natasha. How are you running lakeFS (e.g. local binary, Docker, etc)? And which version?
n

Natasha Taylor

03/08/2023, 11:56 AM
im running with docker via codespaces using the latest version
In the terminal window this error is included in the script "level=warning msg="Failed to get Azure subscription ID from instance metadataGet ...." I'm trying to connect lake fs with an azure storage account so maybe this is the issue?
y

Yoni Augarten

03/08/2023, 12:09 PM
Hey @Natasha Taylor, I apologize for the inconvenience. Since some bugs around the setup page have recently been fixed, can you make sure that the latest lakeFS image is pulled?
n

Natasha Taylor

03/08/2023, 1:40 PM
hi @Yoni Augarten when i run the code im running this which im assuming is pulling the latest lakeFS image
g

Guy Hardonag

03/08/2023, 1:55 PM
Hi @Natasha Taylor, try replacing the docker image tag to a specific version (i.e replace `treeverse/lakefs:latest`` with
treeverse/lakefs:0.95.0
)
r

Robin Moffatt

03/08/2023, 2:41 PM
(not to take this thread off topic, but I wonder if we should update https://docs.lakefs.io/deploy/azure.html#storage-account-access to also include
--pull always
in the
docker run
like we have for the quickstart)
y

Yoni Augarten

03/08/2023, 2:42 PM
Since this one is a production deployment guide, I'm not sure. On the other hand, using "latest" in a production guide is also not the best idea probably.
n

Natasha Taylor

03/08/2023, 2:48 PM
Hi so it's still giving me the same error after changing to treeverse/lakefs:0.95.0 In the terminal window i get this - would it be anything to do with this or is this a different issue?
level=warning msg="Failed to get Azure subscription ID from instance metadataGet \
👀 1
b

Barak Amar

03/08/2023, 3:13 PM
@Natasha Taylor this warn message probably is not the root cause of the failure. Can you list the complete log (if possible) or errors you see? Also what is the UI behaviour - refresh the browser are you still in the setup page?
n

Natasha Taylor

03/08/2023, 3:27 PM
level=warning msg="Failed to get Azure subscription ID from instance metadataGet \"XXX": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" func="pkg/cloud/azure.(*MetadataProvider).GetMetadata" file="build/pkg/cloud/azure/metadata.go:45" error="Get \"XXX": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Yes if i refresh i'm still in the setup page
b

Barak Amar

03/08/2023, 3:33 PM
1. no other messages in the output? 2. from the warning it looks like it fails to communicate with your azure account did the previous setup use azure blob storage?
n

Natasha Taylor

03/08/2023, 3:36 PM
auth.ui_config.login_failed_message="The credentials don't match."
There is this in the output which could be related! Although i've triple checked my credentials and they are correct
b

Barak Amar

03/08/2023, 3:43 PM
will it be possible to upload the full output of the server - without any sensitive information?
s

Sergio Moreta

08/06/2023, 4:10 AM
Hi Natasha and Barak, Have you been able to figure this out? I'm trying to run LakeFS in an Azure Container Instance and connecting to Azure Datalake Gen2 blob storage and think I'm having a similar issue... I'm getting the same Metadata timeout and related warning about not being able to fetch the Azure subscription ID on start-up. Then, whenever I try to create a repo in the LakeFS web UI, I get below nil pointer dereference error. Thanks in advance for any assistance or info provided! 2023/08/06 040008 http: panic serving <MY-IP>64413 runtime error: invalid memory address or nil pointer dereference goroutine 1285 [running]: net/http.(*conn).serve.func1() /usr/local/go/src/net/http/server.go:1850 +0xbf panic({0x1d48820, 0x63f5ac0}) /usr/local/go/src/runtime/panic.go:890 +0x262 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/internal/base.InnerClient[...](0x100?) /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.6.1/internal/base/clients.go:21 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/service.(*Client).generated(...) /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.6.1/service/client.go:111 github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/service.(*Client).NewContainerClient(0xc0004a53e0?, {0xc0071e8132?, 0x0?}) /go/pkg/mod/github.com/!azure/azure-sdk-for-go/sdk/storage/azblob@v0.6.1/service/client.go:126 +0x3f github.com/treeverse/lakefs/pkg/block/azure.(*ClientCache).NewContainerClient.func1() /build/pkg/block/azure/client_cache.go:56 +0x90 github.com/puzpuzpuz/xsync.(*MapOf[...]).doStore(0xc0003c92d0, {0xc0076d1890, 0x15}, 0xc007e3e2e8, 0x1) /go/pkg/mod/github.com/puzpuzpuz/xsync@v1.5.2/mapof.go:238 +0x63e github.com/puzpuzpuz/xsync.(*MapOf[...]).LoadOrCompute(0x5?, {0xc0076d1890?, 0x2?}, 0x2?) /go/pkg/mod/github.com/puzpuzpuz/xsync@v1.5.2/mapof.go:161 +0x28 github.com/treeverse/lakefs/pkg/block/azure.(*ClientCache).NewContainerClient(0xc0004a53e0, {0xc0071e8108, 0x14}, {0xc0071e8132, 0x0}) /build/pkg/block/azure/client_cache.go:50 +0x156 github.com/treeverse/lakefs/pkg/block/azure.(*Adapter).Download(0xc0007fadb0, {0x5565290, 0xc0077e6540}, {{0xc0071e8100, 0x33}, {0x20af1b4, 0x5}, 0x1}, 0x0, 0x0) /build/pkg/block/azure/adapter.go:285 +0x169 github.com/treeverse/lakefs/pkg/block/azure.(*Adapter).Get(0x14?, {0x5565290, 0xc0077e6540}, {{0xc0071e8100, 0x33}, {0x20af1b4, 0x5}, 0x1}, 0xc0075d4600?) /build/pkg/block/azure/adapter.go:195 +0x145 github.com/treeverse/lakefs/pkg/api.(*Controller).ensureStorageNamespace(0xc00721b500, {0x5565290, 0xc0077e6540}, {0xc0071e8100, 0x33}) /build/pkg/api/controller.go:1620 +0xcf github.com/treeverse/lakefs/pkg/api.(*Controller).CreateRepository(0xc00721b500, {0x55632d0, 0xc0073d2090}, 0xc0075d4600, {0xc007d48140, {0xc0008521a0, 0xc}, 0xc00085213c, {0xc0071e8100, 0x33}}, ...) /build/pkg/api/controller.go:1542 +0x572 github.com/treeverse/lakefs/pkg/api.(*ServerInterfaceWrapper).CreateRepository.func1({0x55632d0?, 0xc0073d2090?}, 0x20a0001?) /build/pkg/api/lakefs.gen.go:21059 +0x65 github.com/treeverse/lakefs/pkg/api.(*ServerInterfaceWrapper).CreateRepository(0xc00753a810, {0x55632d0, 0xc0073d2090}, 0xc0075d4500) /build/pkg/api/lakefs.gen.go:21066 +0x5b0 net/http.HandlerFunc.ServeHTTP(0xc007b41970?, {0x55632d0?, 0xc0073d2090?}, 0xc0077e6330?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/treeverse/lakefs/pkg/api.MetricsMiddleware.func1.1({0x7f0be1471348?, 0xc0077e6330}, 0x5563300?) /build/pkg/api/metrics_middleware.go:27 +0xf3 net/http.HandlerFunc.ServeHTTP(0x5563300?, {0x7f0be1471348?, 0xc0077e6330?}, 0xc0075d4500?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x5563300?, 0xc007d7c620?}, 0xc0075d4500) /go/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:147 +0xc5 net/http.HandlerFunc.ServeHTTP(0x5565290?, {0x5563300?, 0xc007d7c620?}, 0x5540fe0?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/treeverse/lakefs/pkg/api.AuthMiddleware.func1.1({0x5563300, 0xc007d7c620}, 0xc00747e800) /build/pkg/api/auth_middleware.go:93 +0x4b0 net/http.HandlerFunc.ServeHTTP(0x1f08860?, {0x5563300?, 0xc007d7c620?}, 0xc?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/treeverse/lakefs/pkg/httputil.DefaultLoggingMiddleware.func1.1({0x5563c00?, 0xc0073f4000}, 0xffffffffffffff01?) /build/pkg/httputil/logging.go:87 +0x645 net/http.HandlerFunc.ServeHTTP(0xc00747e500?, {0x5563c00?, 0xc0073f4000?}, 0xc0074f8c50?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/treeverse/lakefs/pkg/api.OapiRequestValidatorWithOptions.func1.1({0x5563c00, 0xc0073f4000}, 0xc00747e400) /build/pkg/api/serve.go:168 +0x2e3 net/http.HandlerFunc.ServeHTTP(0xc007b21f80?, {0x5563c00?, 0xc0073f4000?}, 0xc0080e8a38?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/go-chi/chi/v5.(*ChainHandler).ServeHTTP(0x1d20d00?, {0x5563c00?, 0xc0073f4000?}, 0xc007170005?) /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.0/chain.go:31 +0x2c github.com/go-chi/chi/v5.(*Mux).routeHTTP(0xc00011d080, {0x5563c00, 0xc0073f4000}, 0xc00747e400) /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.0/mux.go:436 +0x1f9 net/http.HandlerFunc.ServeHTTP(0x55651e8?, {0x5563c00?, 0xc0073f4000?}, 0x63f54b0?) /usr/local/go/src/net/http/server.go:2109 +0x2f github.com/go-chi/chi/v5.(*Mux).ServeHTTP(0xc00011d080, {0x5563c00, 0xc0073f4000}, 0xc007495b00) /go/pkg/mod/github.com/go-chi/chi/v5@v5.0.0/mux.go:87 +0x32a github.com/treeverse/lakefs/cmd/lakefs/cmd.glob..func8.3({0x5563c00, 0xc0073f4000}, 0xc0077d86f0?) /build/cmd/lakefs/cmd/run.go:325 +0x102 net/http.HandlerFunc.ServeHTTP(0x0?, {0x5563c00?, 0xc0073f4000?}, 0x866854?) /usr/local/go/src/net/http/server.go:2109 +0x2f net/http.serverHandler.ServeHTTP({0x555e340?}, {0x5563c00, 0xc0073f4000}, 0xc007495b00) /usr/local/go/src/net/http/server.go:2947 +0x30c net/http.(*conn).serve(0xc007c47b80, {0x5565290, 0xc008102ff0}) /usr/local/go/src/net/http/server.go:1991 +0x607 created by net/http.(*Server).Serve /usr/local/go/src/net/http/server.go:3102 +0x4db
i

Itai Admi

08/06/2023, 7:53 AM
Hey @Sergio Moreta, what’s the lakeFS version you’re using? I’ll try to reproduce it.
s

Sergio Moreta

08/06/2023, 12:38 PM
Hi @Itai Admi, Thanks for the quick response and for offering to try to reproduce it! I'm pulling the latest version from Docker hub. Please let me know if you need any additional details of my deployment (attempt 😉).
i

Itai Admi

08/06/2023, 12:50 PM
Sometime the image
repo/image:latest
gets cached and you don’t run the actual latest version. Can you confirm it’s
0.104.0
?
s

Sergio Moreta

08/07/2023, 12:53 PM
Hi @Itai Admi, Apologies for the delay. Indeed, I'm running version 0.104.0 Thanks again.
j

Jonathan Rosenberg

08/07/2023, 2:58 PM
Hi @Sergio Moreta, Can you please share the lakeFS configurations used to run the server and the format of the blob storage URI you used to create your repository? (hiding sensitive information of course) And just to be sure, the container that you use for the repository creation is under the storage account that you provided in the configurations of lakeFS, correct?
s

Sergio Moreta

08/07/2023, 3:49 PM
Hi @Jonathan Rosenberg, Okay, let me try... For the LakeFS configuration, the only information I'm providing to the container instance is through these environment variables: LAKEFS_DATABASE_TYPE="postgres" LAKEFS_BLOCKSTORE_TYPE="azure" LAKEFS_BLOCKSTORE_AZURE_STORAGE_ACCOUNT="<myblobstorageaccountname>" LAKEFS_DATABASE_POSTGRES_CONNECTION_STRING="host=<mydbname>.postgres.database.azure.com user=<myuser>@<myinstancename>.postgres.database.azure.com password=************" LAKEFS_AUTH_ENCRYPT_SECRET_KEY="<randomkey>" LAKEFS_BLOCKSTORE_AZURE_STORAGE_ACCESS_KEY="<myblobdatacontributorpasswd>" For the URI that I'm providing in the "Create New Repository" window (just before the error) is: https://<myblobstorageaccountname>.blob.core.windows.net/raw/ Lastly, yes, indeed, I've only setup a single storage account, hence both instances of the token <myblobstorageaccountname> refer to the same name. Thank you.
j

Jonathan Rosenberg

08/07/2023, 4:00 PM
Thanks I’ll take a look
👍 1
Hi again, Just to make sure (once more 🙂), is the
LAKEFS_BLOCKSTORE_AZURE_STORAGE_ACCESS_KEY
value you provide an access key to the storage account (that can be found under
Access Keys
under the storage account)? BTW, if you’re deploying lakeFS in Azure, you might want to try to use identity based authentication instead of of the storage account credentials.
s

Sergio Moreta

08/09/2023, 12:58 AM
Hey Jonathan, Thank you for the tips. It indeed seems that I was confusing the two authentication methods and was hence using the wrong key. It looks like I'm one step further now; The web GUI now reports the more verbose message "failed to create repository: failed to access storage" and the nil pointer dereference error is no longer being triggered. I'll also try switching to identity based authentication, as you suggest, since this is now clearly an authentication issue against my storage account. B.t.w. I see that the latest image pulled is now 0.105.0 and noticed some changes in the GUI. Thanks again for the kind assistance.
j

Jonathan Rosenberg

08/09/2023, 3:51 AM
Glad to be of help. If you’re still facing issues, please let us know 🙏
6 Views