Hello everyone, I have some questions about the [M...
# help
f
Hello everyone, I have some questions about the [Multipart Uploads](https://docs.lakefs.io/reference/s3.html): • Are there any best practices about multipart uploads in lakefs? Such as how to call the interfaces in Golang? • Are AbortMultipartUpload and ListParts API available? I encountered some errors when I used the aws-sdk-go-v2 to call the interfaces.
o
hey @fuzhy let me check that for you
@fuzhy to your question, multipart uploads are supported through the S3 gateway and handled automatically when you interact with lakeFS using S3 gateway. AbortMultipartUpload is executed automatically when you interact with the DeleteObject operation. I noticed that the documentation says the we support the ListParts API but I couldn’t find a reference to it in the code, I’ll open an issue to either add support or remove it from our documentation. Do you mind explaining your use-case and what errors you’ve encountered?
f
@Or Tzabary Thank you for your reply! I have an API Server with lakeFS as the underlying git-like storage solution and the Server provides the APIs for multipart uploads. I tried calling the CreateMultipartUpload/Upload Part/CompleteMultipartUpload API with aws-sdk-go-v2 and they all worked. The error I encountered when calling AbortMultiPartUpload:
Copy code
level=error msg="could not abort multipart upload" func="pkg/gateway/operations.(*DeleteObject).HandleAbortMultipartUpload" file="build/pkg/gateway/operations/deleteobject.go:36" error="NoSuchUpload: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.\n\tstatus code: 404, method=DELETE operation_id=delete_object service_name=s3_gateway upload_id=xxx user=admin
o
I see, sorry to hear that... I think that the AbortMultipart is implemented, I'll double check, if not, I'll include that in the issue too. Will update
f
Yeah, I also saw the implementation of gateway about AbortMultipart, and the s3 blockstore also implements the AbortMultiPartUpload of the Adapter interface. Some tests may need to be added like local blockstore... and it would be better if there are some usage examples.
👍 1
o
the S3 gateway does implement
AbortMultiPartUpload
, though the error message you shared seems to be related to something else, it looks like the
DeleteObject
handler was triggered with
UploadId
(which triggers the
AbortMultiPartUpload
and it was invalid). I’ve added to the issue request to add examples, but to better understand what went wrong for you, can you please share what you tried to do with the S3 gateway?
f
The code is as follows:
Copy code
input := &s3.AbortMultipartUploadInput{
		Bucket:   aws.String(repo),
		Key:      aws.String(filepath.Join(branch, prefix)),
		UploadId: aws.String(uploadId),
	}
	// client is the aws s3 client from the provided lakefs address/accessKeyID/secretAccessKey
	output, err := client.AbortMultipartUpload(ctx, input)
o
And how do you know the multipart information? (UploadId for instance)
f
the error message:
Copy code
operation error S3: AbortMultipartUpload, exceeded maximum number of attempts, 3, https response error StatusCode: 500, api error InternalError: We encountered an internal error, please try again.
And the previous error information is the log of lakefs.
I have a correct call to
CreateMultipartUpload
and it return the uploadId. I can also call
UploadPart
and
CompleteMultipartUpload
with the uploadId.
o
Thank you for the information, I'll try to reproduce it on myself to see if I can find the root cause or if there's any workaround. May I ask what's your use case? What are you trying to do?
f
BTW, the version of lakefs is 0.94.1 and the blockstore type is s3 based minio. Hope this information is helpful to you.
o
I managed to reproduce it myself using lakeFS 0.97.4 with S3 backed storage
can you please report a bug describing this issue?
f
OK, I will open an issue later. Thank you for your patience and careful guidance.
o
happy to help, please let me know if there’s anything else I can help with
gratitude thank you 1
f
nothing else, thank you
jumping lakefs 1
196 Views