Or is it how LakeFS normalizes the S3 headers and then signs it?

I'm not sure, but I think the problem comes from how S3 (and lakeFS) build the canonical request for sigv4 signing. They normalize header values (trim and collapse spaces) before computing the signature. So if a client signs the raw value with double spaces, but lakeFS normalizes it to single spaces before verifying, we get

SignatureDoesNotMatch

. In short, it's not an RFC limitation, it's the sigv4 canonicalization step that causes the mismatch.

Hmm so perhaps object-store-rs sigv4 canonicalization is non-standard

Thanks, great work! To add links to Ben-El's message, I believe that a single space is explicitly required by this section, specifically the explanation of how to transform headers to

CanonicalHeaders

. Could you please connect us directly with object-store-rs if thing it's a bug on our side, so we don't need to go through you?

@Ariel Shaqed (Scolnicov) I only asked in the discord server of arrow-rs otherwise I can open a thread in the rust repo so we can continue discussion there?

Hi @Ion, Thanks! If discussion is needed, I'm probably happier to do so on an issue tracker than on a Discord channel. But really, whatever works for the object-store-rs and/or arrow-rs people.

Thanks! I actually now believe that this used to be a bug in lakeFS 🫢 😱 and we fixed it: the spec has this weird canonicalization requirement "convert sequential spaces to a single space." for values. Let me hash it out on that issue, and then we will see how to proceed.