Hey lakesfs team, curious to get your thoughts on ...
# helpt
Thomas Vander Wal
09/03/2021, 3:26 AMHey lakesfs team, curious to get your thoughts on this issue I just opened. I believe there is a bug with adding data to LakeFS when SigV4 signing is used.
Thomas Vander Wal
09/03/2021, 3:27 AMI didn’t test with any other client using that signing mechanism but could be more than RClone.
a
Ariel Shaqed (Scolnicov)
09/03/2021, 3:32 AMHi @Thomas Vander Wal,
Thanks for the detailed report!
Could you please check whether the s3 gateway on your lakeFS is set to the correct region?
Ariel Shaqed (Scolnicov)
09/03/2021, 3:34 AMrclone appears to be using us-gov-west-1.
SigV4 signs the intended region, which can cause such hard to debug issues.
t
Thomas Vander Wal
09/03/2021, 3:36 AMNo problem! Happy to contribute 🙂
Yea I verified that region matches what I have in the gateway
a
Ariel Shaqed (Scolnicov)
09/03/2021, 4:14 AMCould you please post the server version onto the bug when you get a chance?
t
Thomas Vander Wal
09/03/2021, 4:26 AMFor sure. I added both server version and rclone version info
👍🏾 1
Thomas Vander Wal
09/03/2021, 6:49 AMHey @Ariel Shaqed (Scolnicov) I saw you message and will try to get those to you over the weekend. If not it might be next week as we don’t work Fridays typically and it’s a holiday in the US.
I can do some testing as well as we might be hitting the same issue. I’m trying to copy an existing S3 bucket of training data so the customer has a versioned set to work with. No files currently exist in LakeFS so rclone would be trying to create them and not overwrite
a
Ariel Shaqed (Scolnicov)
09/03/2021, 6:54 AMHowever I did not get 403s but rather other errors, mostly 404s and unsupported operations (rclone decided to try to change bucket configuration). Note that I know next to nothing about rclone, operator error on my part is very possible.
Ariel Shaqed (Scolnicov)
09/03/2021, 6:54 AMAlso no worries about the weekend; enjoy it!
t
Thomas Vander Wal
09/05/2021, 5:35 AM@Ariel Shaqed (Scolnicov) Ok I added those to the ticket. Also I ran into another weird issue on trying to load data in.
Normal files work fine with the auth v2 but as soon as there is a Unicode character in the file name I get that signature mismatch error… rclone doesn’t seem to treat those any differently and the traffic it’s correctly URL encoding it. I added a comment to the issue with some more specifics on that
Thomas Vander Wal
09/05/2021, 5:45 AMI noticed it flash by too on a multipart issue, so it seems like LakeFS could erroring with that auth message when it’s not actually an auth issue.
a
Ariel Shaqed (Scolnicov)
09/05/2021, 5:49 AMWow, did not think to test with Unicode. Let me take another look...
t
Thomas Vander Wal
09/05/2021, 6:03 AMThanks Ariel! I know these are some odd edge cases and I appreciate you taking the time to dig in.
2 Views