HI peeps! New to this slack community so short introduction. I ran into Oz Katz last week at the Databricks Data & AI Summit in San Francisco, slightly after his talk, and am now looking into how we could use LakeFS. I have a small use-case in mind that mind benefit from git-style versioning on a group of tables. We're in Azure, however, and in the (open source) code within the file pkg/cloud/azure/metadata.go there is a reference to an internal IP adress to fetch Azure Metadata. This fails and it throws an error, saying it cannot find the sub. I did a reverse lookup, and it seems to be a private IP. So, what's going on here exactly? :)

Hi @tom kennes, welcome to the lake! I’m not an expert on this, but does this help?

That only works if you're running LakeFS on a VM within Azure right? And even then you would need to be able to access that endpoint. Sounds a bit cumbersome to get the subscription. Why not take the Azure Subscription into account as a user argument? In the end you only use it to find the azure blob right?

I think that this shouldn’t create an issue for you running lakeFS on azure (since multiple folks run it on their azure environments, mostly not on VMs). Can you please share your stack trace?

WARNING[2024-06-24T084053Z]build/pkg/cloud/azure/metadata.go:45 pkg/cloud/azure.(*MetadataProvider).GetMetadata Failed to get Azure subscription ID from instance metadataGet "http://169.254.169.254/metadata/instance?api-version=2019-03-11&format=json": context deadline exceeded (Client.Timeout exceeded while awaiting headers) error="Get \"http://169.254.169.254/metadata/instance?api-version=2019-03-11&format=json\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Hey @tom kennes, that request is triggered in a backoffice flow and is part of the mechanism we have in place in the server to check for new versions of lakeFS, so that we can notify the user for upgrades and CVEs. Failures in this side flow do not impact anything else in lakeFS.

Thanks! That's fast. Are you refering to the first or the last error message?

Both 🙂 The second is because your code doesn't trust the certificate provided on the audit endpoint, the first is because it's not running inside Azure Cloud so it cannot hit the cloud metadata "server" to know what is the instance. Neither blocks you and both are safe to ignore! You will not see upgrade notifications on startup, so you will just need to be slightly more aware of new versions coming out.

Ah right, it's not meant to be run outside of Azure, and I need to make sure to whitelist that IP Adress

You’d still be safe. In the context of the check, lakeFS doesn’t need more than a subscription ID, and it doesn’t need to match your storage subscription ID.