Hello everyone I have some questions about the Multipart Upl lakeFS #help

Hello everyone, I have some questions about the [M...

fuzhy

04/20/2023, 12:03 PM

Hello everyone, I have some questions about the [Multipart Uploads](https://docs.lakefs.io/reference/s3.html): • Are there any best practices about multipart uploads in lakefs? Such as how to call the interfaces in Golang? • Are AbortMultipartUpload and ListParts API available? I encountered some errors when I used the aws-sdk-go-v2 to call the interfaces.

Or Tzabary

04/20/2023, 12:11 PM

hey @fuzhy let me check that for you

Or Tzabary

04/20/2023, 12:30 PM

@fuzhy to your question, multipart uploads are supported through the S3 gateway and handled automatically when you interact with lakeFS using S3 gateway. AbortMultipartUpload is executed automatically when you interact with the DeleteObject operation. I noticed that the documentation says the we support the ListParts API but I couldn’t find a reference to it in the code, I’ll open an issue to either add support or remove it from our documentation. Do you mind explaining your use-case and what errors you’ve encountered?

fuzhy

04/21/2023, 2:56 AM

@Or Tzabary Thank you for your reply! I have an API Server with lakeFS as the underlying git-like storage solution and the Server provides the APIs for multipart uploads. I tried calling the CreateMultipartUpload/Upload Part/CompleteMultipartUpload API with aws-sdk-go-v2 and they all worked. The error I encountered when calling AbortMultiPartUpload:

Copy code

level=error msg="could not abort multipart upload" func="pkg/gateway/operations.(*DeleteObject).HandleAbortMultipartUpload" file="build/pkg/gateway/operations/deleteobject.go:36" error="NoSuchUpload: The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.\n\tstatus code: 404, method=DELETE operation_id=delete_object service_name=s3_gateway upload_id=xxx user=admin

Or Tzabary

04/21/2023, 3:09 AM

I see, sorry to hear that... I think that the AbortMultipart is implemented, I'll double check, if not, I'll include that in the issue too. Will update

fuzhy

04/21/2023, 3:22 AM

Yeah, I also saw the implementation of gateway about AbortMultipart, and the s3 blockstore also implements the AbortMultiPartUpload of the Adapter interface. Some tests may need to be added like local blockstore... and it would be better if there are some usage examples.

👍 1

Or Tzabary

04/21/2023, 5:23 AM

the S3 gateway does implement

AbortMultiPartUpload

, though the error message you shared seems to be related to something else, it looks like the

DeleteObject

handler was triggered with

UploadId

(which triggers the

AbortMultiPartUpload

and it was invalid). I’ve added to the issue request to add examples, but to better understand what went wrong for you, can you please share what you tried to do with the S3 gateway?

fuzhy

04/21/2023, 5:40 AM

The code is as follows:

Copy code

input := &s3.AbortMultipartUploadInput{
		Bucket:   aws.String(repo),
		Key:      aws.String(filepath.Join(branch, prefix)),
		UploadId: aws.String(uploadId),
	}
	// client is the aws s3 client from the provided lakefs address/accessKeyID/secretAccessKey
	output, err := client.AbortMultipartUpload(ctx, input)

Or Tzabary

04/21/2023, 5:41 AM

And how do you know the multipart information? (UploadId for instance)

fuzhy

04/21/2023, 5:42 AM

the error message:

Copy code

operation error S3: AbortMultipartUpload, exceeded maximum number of attempts, 3, https response error StatusCode: 500, api error InternalError: We encountered an internal error, please try again.

And the previous error information is the log of lakefs.

fuzhy

04/21/2023, 5:45 AM

I have a correct call to

CreateMultipartUpload

and it return the uploadId. I can also call

UploadPart

and

CompleteMultipartUpload

with the uploadId.

Or Tzabary

04/21/2023, 5:47 AM

Thank you for the information, I'll try to reproduce it on myself to see if I can find the root cause or if there's any workaround. May I ask what's your use case? What are you trying to do?

fuzhy

04/21/2023, 5:50 AM

BTW, the version of lakefs is 0.94.1 and the blockstore type is s3 based minio. Hope this information is helpful to you.

Or Tzabary

04/21/2023, 5:51 AM

I managed to reproduce it myself using lakeFS 0.97.4 with S3 backed storage

Or Tzabary

04/21/2023, 5:51 AM

can you please report a bug describing this issue?

fuzhy

04/21/2023, 5:56 AM

OK, I will open an issue later. Thank you for your patience and careful guidance.

Or Tzabary

04/21/2023, 5:56 AM

happy to help, please let me know if there’s anything else I can help with

gratitude thank you 1

fuzhy

04/21/2023, 5:58 AM

nothing else, thank you

jumping lakefs 1

376 Views

Open in Slack

Previous Next