Hello, I deployed a LakeFS OSS instance to Google Cloud Run for evaluation using this guide: https://docs.lakefs.io/howto/deploy/gcp.html I was testing it out with a dataset that is 1.1G and has ~7.5k files within it using

lakectl local

. The basic steps were: •

lakectl local init <lakefs://repo/branch/path> ./local-path

•

lakectl local commit local-path -m 'testing this out'

The commit command then runs for a few seconds, but errors out with:

upload folder/example.pdf failed: get physical address to upload object: http2: response body closed

and the whole thing stops. Any guidance on how to resolve this issue?

Hi @Aaron Taylor! Sorry to hear you are having problems with lakeFS. Can you tell me please, what is the size of example.pdf? Also, can you see if you have any errors on the lakeFS server side?

It is a different file each time, when I ran it just now it was an Excel file that was

344K

which it failed on. And ah, looking at the logs I see a lot of this:

time="2024-08-22T02:47:25Z" level=error msg="error generating pre-signed URL" func="pkg/block/gs.(*Adapter).GetPreSignedURL" file="build/pkg/block/gs/adapter.go:239" error="unable to sign bytes: googleapi: Error 403: Permission 'iam.serviceAccounts.signBlob' denied on resource (or it may not exist).\nDetails:\n[\n  {\n    \"@type\": \"type.googleapis.com/google.rpc.ErrorInfo\",\n    \"domain\": \"iam.googleapis.com\",\n    \"metadata\": {\n      \"permission\": \"iam.serviceAccounts.signBlob\"\n    },\n    \"reason\": \"IAM_PERMISSION_DENIED\"\n  }\n]" host=<snipped> method=GET operation_id=GetPhysicalAddress path="<snipped out of this example>&presign=true" user=plus-dev

I'm guessing that may be the cause? I saw the protocol error in the client and had assumed it was going wrong at the network layer somehow instead. I will look at granting permissions to sign blobs and see if that fixes it!

Seems like this is the issue. As a WA and to verify lakectl works you can pass --pre-sign=false to the local commands

oh great, seems like that is working! Thank you. Are there any docs on the specific IAM roles that LakeFS requires for a GCP deployment?

Glad to hear, indeed we need to update the GCP docs for the presign support