Hi Not sure if this is an issue with LakeFS Rclone or someth

Question

Hi Not sure if this is an issue with LakeFS Rclone or something on my side I m using rclone to copy from one lakefs instance to another like this rclone sync checksum s3 use multipart etag=true lakefs lt a repo1 tag1 gt lakefs lt b repo1 main gt But I m getting an error from rclone Failed to copy multipart upload corrupted Etag differ expecting 1234567890abcdef1234567890abcdef 51 but got 1234567890abcdef1234567890abcdef In other words it s comparing a 32 character etag including appended chunk count to the same etag withOUT the appended chunk count Of course they don t match This happens if the file that s being copied is above the size threshold for multipart uploads If I drop the s3 use multipart etag=true parameter it doesn t fail but I m not sure how much validation is done on the copy When I do lakectl fs stat on the file on lakefs a the Checksum includes the appended chunk count

Iddo Avneri · Answer

Thanks for sharing Chuck

Iddo Avneri · Answer

What is your rclone version

Iddo Avneri · Answer

Let me consult with some teammates regarding this I think some information that can be helpful is the versions rclone lakectl so we can try and look at the specific configuration when we investigate

CC · Answer

LakeFS client is 0 86 0 rclone version 1 60 0

CC · Answer

The rclone config file is lakefs a type = s3 provider = Other env auth = false access key id = REMOVED secret access key = REMOVED endpoint = no check bucket = true lakefs b type = s3 provider = Other env auth = false access key id = REMOVED secret access key = REMOVED region = us east 1 endpoint = no check bucket = true

Iddo Avneri · Answer

Very helpful Thank you

CC · Answer

Is provider = Other kosher Your examples tend to say provider = AWS but rclone errors out with that possibly because 2 different regions are involved

Iddo Avneri · Answer

I m honestly not 100 sure and will consult with Ariel I think m this is sufficient information to look into it

Ariel Shaqed (Scolnicov) · Answer

Thanks < CC> for the excellent report and all the accompanying information I am investigating S3 multipart ETags are wonky and it can be frustrating to track ETags on S3 compatible object stores rclone actually it has this this this etc lakeFS copies ETags during import so it makes sense that imported objects share the same ETag string Appending the chunk count to the computed ETag is a really strange move for a checksum I will check 1 Can we include the chunk count on S3 multipart uploads 2 Assuming we can will it be safe to do so for our existing users I should have at least a plan by 16 00 UTC today As a workaround if your repo import is not very large You might use Rclone as import too It should add the magic `X Amz Meta Md5chksum` tag on lakefs a and then further Rclones should work just fine

Ariel Shaqed (Scolnicov) · Answer

Actually I m clearly not an Rclone expert But from reading the above issues and browsing the code I suspect that Rclone cannot handle ETags that arise from multipart uploads to S3 First some code Rclone ignores any ETag that does not match ```var matchMd5 = regexp MustCompile `^ 0 9a f 32 $` ``` so it will refuse to handle an ETag generated by a multipart upload So I came up with this experiment It appears to show that Rclone does not verify ETags for S3 copies No lakeFS in sight I started by uploading 10 MiB of data to my US bucket ``` aws s3 cp tmp 10m ``` Let s rclone sync it over to my EU bucket ` vv` means to give lots of debug information aws s3api head object bucket treeverse ariels test key test rclone 10m 10m AcceptRanges bytes LastModified Thu 12 Jan 2023 08 26 03 GMT ContentLength 10485760 ETag 895bed95bff6a288b45c28697991cc08 ContentType binary octet stream Metadata ``` rclone sync vv checksum s3 use multipart etag=true lt 7 gt DEBUG rclone Version 1 60 1 starting with parameters rclone sync vv checksum s3 use multipart etag=true lt 7 gt DEBUG rclone systemd logging support activated lt 7 gt DEBUG Creating backend with remote lt 7 gt DEBUG Using config file from home ariels config rclone rclone conf lt 7 gt DEBUG s3 detected overridden config adding 5gJq8 suffix to name lt 7 gt DEBUG fs cache adding new entry for parent of s3 5gJq8 treeverse ariels test us test rclone lt 7 gt DEBUG Creating backend with remote lt 7 gt DEBUG s3 detected overridden config adding 5gJq8 suffix to name lt 5 gt NOTICE S3 bucket treeverse ariels test path test rclone Switched region to eu central 1 from us east 1 lt 7 gt DEBUG pacer low level retry 1 2 error BucketRegionError incorrect region the bucket is not in us east 1 region at endpoint bucket is in eu central 1 region status code 301 request id 64Y2G17VXMKHFPA6 host id lmJhpdYGGgFn10pdzK2 oiF4jY7MVqVolaE7 lNzpvz+AQZddc158KkisiuXnyEV7UsNlZM8zQU= lt 7 gt DEBUG pacer Rate limited increasing sleep to 10ms lt 7 gt DEBUG pacer Reducing sleep to 0s lt 7 gt DEBUG fs cache renaming cache item to be canonical s3 5gJq8 treeverse ariels test test rclone 10m lt 7 gt DEBUG 10m Need to transfer File not found at Destination lt 7 gt DEBUG 10m Src hash empty aborting Dst hash check lt 6 gt INFO 10m Copied server side copy lt 6 gt INFO Transferred 10 MiB 10 MiB 100 0 B s ETA Transferred 1 1 100 Elapsed time 6 5s lt 7 gt DEBUG 11 go routines active``` The key line is `10m Src hash empty aborting Dst hash check` Indeed as expected the US object has a multipart upload ETag suffix 2 10 megs is 2 parts because a part is 5 MiB and no auxiliary Rclone metadata because it was generated by `aws cp` not `rclone` ``` aws s3api head object bucket treeverse ariels test us key test rclone 10m AcceptRanges bytes LastModified Thu 12 Jan 2023 08 21 41 GMT ContentLength 10485760 ETag e790610b43d3959ea3474ee6097ee2b4 2 ContentType binary octet stream Metadata ``` Over on the other bucket we have a different ETag presumably because rclone asked S3 to perform a server side copy ``` aws s3api head object bucket treeverse ariels test key test rclone 10m 10m AcceptRanges bytes LastModified Thu 12 Jan 2023 08 26 03 GMT ContentLength 10485760 ETag 895bed95bff6a288b45c28697991cc08 ContentType binary octet stream Metadata ``` Server side copy changes ETags see for instance I also tried to copy to a local minio on my machine which prevents server side copy Again it claims no hash check ``` rclone sync vv checksum s3 use multipart etag=true lt 7 gt DEBUG rclone Version 1 60 1 starting with parameters rclone sync vv checksum s3 use multipart etag=true lt 7 gt DEBUG rclone systemd logging support activated lt 7 gt DEBUG Creating backend with remote lt 7 gt DEBUG Using config file from home ariels config rclone rclone conf lt 7 gt DEBUG s3 detected overridden config adding 5gJq8 suffix to name lt 7 gt DEBUG fs cache adding new entry for parent of s3 5gJq8 treeverse ariels test us test rclone lt 7 gt DEBUG Creating backend with remote lt 7 gt DEBUG minio detected overridden config adding 5gJq8 suffix to name lt 7 gt DEBUG fs cache renaming cache item to be canonical minio 5gJq8 foo test rclone lt 7 gt DEBUG 10m Need to transfer File not found at Destination lt 7 gt DEBUG 10m Src hash empty aborting Dst hash check lt 6 gt INFO 10m Copied new lt 6 gt INFO Transferred 10 MiB 10 MiB 100 496 503 KiB s ETA 0s Transferred 1 1 100 Elapsed time 20 5s lt 7 gt DEBUG 7 go routines active``` I am sorry I see no way forward for us here until we understand how to make Rclone verify hashes

Ariel Shaqed (Scolnicov) · Answer

Final message for today I promise This is not just checksum checking to see if sync is needed I also tried `rclone check` For 2 distinct files I get ``` rclone check lt 5 gt NOTICE S3 bucket treeverse ariels test us path test rclone 2 0 differences found lt 5 gt NOTICE S3 bucket treeverse ariels test us path test rclone 2 1 hashes could not be checked lt 5 gt NOTICE S3 bucket treeverse ariels test us path test rclone 2 1 matching files``` And for 2 identical files I get the same output ``` rclone check s3eu test rclone lt 5 gt NOTICE S3 bucket treeverse ariels test path test Switched region to eu central 1 from us east 1 lt 5 gt NOTICE S3 bucket treeverse ariels test path test rclone 0 differences found lt 5 gt NOTICE S3 bucket treeverse ariels test path test rclone 1 hashes could not be checked lt 5 gt NOTICE S3 bucket treeverse ariels test path test rclone 1 matching files``` Really sorry but if we cannot figure out how to do it on S3 I will not able to do it on lakeFS either

CC · Answer

Thanks for that research I see several takeaways 1 Using rclone across the board may be the answer I need 2 Using vv for a couple of simple tests should at least tell me if it s using the hash 3 From what I ve seen on the web I gather that rclone is using checksums on each chunk of the uploads even if it s not doing it for them as a set but that isn t an option for downloads which weakens the net result of course

Ariel Shaqed (Scolnicov) · Answer

Yup The whole situation with Rclone and S3 and ETag is deeply unsatisfying at an engineering level