Write back from duckDB to lakeFS I get an ambiguou...
# helpr
Robin Moffatt
03/03/2023, 6:02 PMWrite back from duckDB to lakeFS I get an ambiguous error when I try to write to a protected branch.
I'm not familiar enough with the interface between the two tools here. Is lakeFS throwing a more verbose exception that duckDB is squashing, or is lakeFS just throwing a generic error that could be better ("Cannot write to a protected branch" for example)
Copy code
D export database '<s3://drones03/main/drone-registrations/>' (format parquet);
94% ▕████████████████████████████████████████████████████████▍ ▏ Error: Invalid Error: IO Error: Unexpected response during S3 multipart upload finalizationb
Barak Amar
03/03/2023, 6:06 PM
S3 had no concept of branch protection. so my suggestion is to find the equivalent error you expect from S3, by executing it against S3. We can align the error to match it.
I assume lakefs current error is generic one (for S3 protocol)
a
Ariel Shaqed (Scolnicov)
03/03/2023, 6:08 PMWhenever there's any doubt, I like to blame S3 clients for deleting all useful information from errors.
On a serious note, I'll take a look on Sunday as to what error message lakefs encodes in these cases. However, as @Barak Amar said, there aren't that many useful S3 error types in the first place.
Ariel Shaqed (Scolnicov)
03/05/2023, 2:38 PMChecking the code, it seems that writing to a protected branch should give you some forbidden code (a 403). Could you check the lakeFS logs and see if we do indeed generate that? If we do, I'll play my "blame the S3 client" card, sorry.
r
Robin Moffatt
03/06/2023, 9:21 AMthanks both. This was against lakeFS cloud so I can't see the logs. I'll see if I can reproduce it locally.
Robin Moffatt
03/06/2023, 11:23 AMlocally I see this:
Copy code
time="2023-03-06T11:22:57Z" level=error msg="could not update metadata" func="pkg/gateway/operations.(*PathOperation).finishUpload" file="build/pkg/gateway/operations/operation_utils.go:51" error="cannot write to protected branch" host="127.0.0.1:8000" matched_host=false method=POST operation_id=post_object path=foo.parquet physical_address=data/gp4jc9og1v5s73bu3j9g/cg2sq48g1v5s73bu3jag ref=main repository=drones03 request_id=6e0d5032-017a-410d-a6e6-fac8ad9bcc66 service_name=s3_gateway upload_id=79fdb595477b4f86ad73fcebb24b3b13 user=admina
Ariel Shaqed (Scolnicov)
03/06/2023, 1:18 PMYeah, that seems fine. I reckon it's some issue in the DuckDB client or some library that it uses, and you lose the error message.
Ariel Shaqed (Scolnicov)
03/06/2023, 1:20 PMUggh 🤢 :
``` auto open_tag_pos = result.find("<CompleteMultipartUploadResult", 0);
if (open_tag_pos == string::npos) {
throw IOException("Unexpected response during S3 multipart upload finalization");
}```
Ariel Shaqed (Scolnicov)
03/06/2023, 1:20 PMThe code loses the error message.
😞 2
r
Robin Moffatt
03/06/2023, 3:07 PM140 Views