I am seeing random failures

SignatureDoesNotMatch

the request signature we calculated does not match the signature you provided. Check your key and signing method.

Any ideas, I found a issue in the repo that also points to boto, but I am using obstore (object-store-rs)

In general, when AWS-style library signatures don't match and your credentials are correct, look for a "region" setting in the library In some libraries, "us-east-1" is assumed but there's a lot of libraries out there. The region is part of the hash/signature algorithm, even though lakefs itself doesn't technically run in a region. You can configure this in the server config if you prefer to change it https://docs.lakefs.io/v1.67/reference/configuration/#gateways

Could also be related that it happens on objects that are 0 bytes?

With the amazon-maintained clients like boto or aws-sdk-go, often you can test this theory by setting the environment variable

AWS_DEFAULT_REGION

but some libraries force you to specify (even when you're overriding the backing URLs) in the constructor

hey @Ion - does it happen when using the S3 gateway? only for 0 byte objects? are you attempting to PutObject or GetObject when it happens?

It happened as far as I saw only on 0 byte objects and using putObjevt