Write back from duckDB to lakeFS I get an ambiguou...
# help
Write back from duckDB to lakeFS I get an ambiguous error when I try to write to a protected branch. I'm not familiar enough with the interface between the two tools here. Is lakeFS throwing a more verbose exception that duckDB is squashing, or is lakeFS just throwing a generic error that could be better ("Cannot write to a protected branch" for example)
Copy code
D export database '<s3://drones03/main/drone-registrations/>' (format parquet);
 94% ▕████████████████████████████████████████████████████████▍   ▏ Error: Invalid Error: IO Error: Unexpected response during S3 multipart upload finalization
It works just fine writing to a branch - I'm just interested in where to follow up re. improving the error
S3 had no concept of branch protection. so my suggestion is to find the equivalent error you expect from S3, by executing it against S3. We can align the error to match it. I assume lakefs current error is generic one (for S3 protocol)
Whenever there's any doubt, I like to blame S3 clients for deleting all useful information from errors. On a serious note, I'll take a look on Sunday as to what error message lakefs encodes in these cases. However, as @Barak Amar said, there aren't that many useful S3 error types in the first place.
Checking the code, it seems that writing to a protected branch should give you some forbidden code (a 403). Could you check the lakeFS logs and see if we do indeed generate that? If we do, I'll play my "blame the S3 client" card, sorry.
thanks both. This was against lakeFS cloud so I can't see the logs. I'll see if I can reproduce it locally.
locally I see this:
Copy code
time="2023-03-06T11:22:57Z" level=error msg="could not update metadata" func="pkg/gateway/operations.(*PathOperation).finishUpload" file="build/pkg/gateway/operations/operation_utils.go:51" error="cannot write to protected branch" host="" matched_host=false method=POST operation_id=post_object path=foo.parquet physical_address=data/gp4jc9og1v5s73bu3j9g/cg2sq48g1v5s73bu3jag ref=main repository=drones03 request_id=6e0d5032-017a-410d-a6e6-fac8ad9bcc66 service_name=s3_gateway upload_id=79fdb595477b4f86ad73fcebb24b3b13 user=admin
Yeah, that seems fine. I reckon it's some issue in the DuckDB client or some library that it uses, and you lose the error message.
Uggh 🤢 :
``` auto open_tag_pos = result.find("<CompleteMultipartUploadResult", 0);
if (open_tag_pos == string::npos) {
throw IOException("Unexpected response during S3 multipart upload finalization");
The code loses the error message.
😞 2