Hey, we are using an S3 compatible bucket, and when we try to upload data with "--direct" flag, we get the following error

upload to backing store: MissingRegion: could not find region configuration

I believe this can be simply solved by setting ``blockstore.s3.region``, but what should we set it to when we are not using AWS directly?

Hi @Matija Teršek, My I ask which s3 compatible provider you are using?

Backblaze. Thanks, will give it a try 🙂

Sure, np

So it does seem we do have this set up already:

LAKEFS_BLOCKSTORE_S3_DISCOVER_BUCKET_REGION=false

You'd also need to set also need to set a few other parameters You can reference this part of the docs It's for on-prem, but should be the same for s3-compatible blockstores

Is this error message from lakectl perhaps? If so you may need to configure the AWS region in an environment variable, probably AWS_REGION but perhaps AWS_DEFAULT_REGION. Basically in direct mode your client environment has to work with the AWS cli.

We do have it set up like in the on-prem installation. So force path, discover region to false, and then connection with S3 storage works. But when using

--direct

in command line causes the error yeah. So for --direct mode, AWS cli is also required?

You're using

lakectl

, right?

Exactly

Sorry for confusing messages. You'll need a client-side configuration that can access AWS directly to use direct mode. There may be a way using the new presigned URLs feature, that won't require any additional client-side configuration. I'll need to get to a keyboard to verify, unless @Barak Amar can beat me to it?!

Hi

Hey, we are using 0.90.1

You will need v0.91.0

Both lakeFS server and lakectl. Brand new.

Testing it out now and benchmarking (it seems to work). I know @Ariel Shaqed (Scolnicov) said this

That passes data directly to s3, and only performs metadata operations on lakeFS.

And the docs say this

write directly to backing store (faster but requires more credentials)

But I'm still a bit confused on what the difference is between using

--direct

/

--pre-sign

vs without? Why should it be faster?

With "regular" upload, the data flow looks something like this: local machine --> lakeFS --> S3 The up side is that you don't need the local machine to have credentials to the backing store and you can still upload files

Great thank you @Elad Lachmi, I understand

Sure, np

So after timing with

--pre-sign

it seems like it still took over 2 hours to upload 2.5 GB

Interesting Thanks for taking the time to benchmark! We'll need to look into that

Looking at the reverse proxy logs, it seems that for each file a link is generated that let's you upload a single file. Not sure if that's possible at all with S3 API, but perhaps it might make more sense to get the link for the entire folder?

Unfortunately I don't know of S3 support for presigned URLs for anything except a single object. Generally presigning is for a particular request, and it cannot be changed. We will of course investigate. If direct uploads are fast, we are probably running into a metadata rate limitation rather than a data rate limitation. Would you feel comfortable sharing any of the following information? • How many files are we uploading? I saw 5000 on some previous conversation, but that sounds really slow so I'm making sure. • Which database are we using for KV: Postgres / DynamoDB / ...? • Where is the lakeFS server located? Where is the database? Where is the bucket? • Could you share the lakeFS config file? We obviously don't need any of the secrets, so anything under auth, secret_access_key, session_token, dynamodb.aws_secret_access_key, encrypt.secret_key. If you can, please send a scrubbed version directly to me.

I saw 5000 on some previous conversation, but that sounds really slow so I'm making sure.

to answer this one - it's 5000 images and 5000 json files, so 10,000 total

Any additional information you can comfortably share would be helpful and we'll look into it

I'll refer to @Matija Teršek for the other bullet points

Bucket is US west, DB and server are both east. We use the following env variables (value not provided where secret)

LAKEFS_DATABASE_POSTGRES_TYPE=postgres
LAKEFS_BLOCKSTORE_TYPE=s3
LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE=true
LAKEFS_BLOCKSTORE_S3_DISCOVER_BUCKET_REGION=false
LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID
LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY
LAKEFS_DATABASE_CONNECTION_STRING
LAKEFS_AUTH_ENCRYPT_SECRET_KEY
LAKEFS_DATABASE_POSTGRES_MAX_IDLE_CONNECTIONS=10
LAKEFS_DATABASE_POSTGRES_MAX_OPEN_CONNECTIONS=22

Hello @Matija Teršek! Thanks for sharing your configs. I’ll let @Conor Simmons and lakeFS team take a look at this.

Bucket is US west, DB and server are both east.

Ouch, this may be an issue: lakeFS is best used when DB, server and bucket are in the same region. It would be ideal if you could test with the bucket and the client in the same region also. If you can, you really should run bucket, DB and server close to one another. Everything from now on assumes that for some reason you cannot. If I had to guess, I would say that the issue is metadata not data. You are uploading 10_000 files IIUC, but only 2.5GiB in total. And that takes you over 2 hours. You're clearly not running out of data throughput anywhere, so I guess the cross-region metadata operations on lakeFS are slowing everything down. Where is the client located? A direct upload requires 2 API calls on lakeFS, and a long-running API call on the bucket. If you're running the client near the bucket on the west coast, each API call is at least the inter-region RTT -- this is on average 62ms (us-east-1 to us-west-1 or us-west-2), and you'll be limited to fewer than 8 uploads per second in the best case. If you need to reconnect the client, that's another round-trip. Ideally you would not be running lakeFS far from the data store. Doing it like this will be slow and expensive because of all the cross-region transfers. If all this is true, one way to speed up uploads would be to perform them massively in parallel. Since lakeFS is not busy in this case, we could easily perform many hundreds of these uploads in parallel, maybe add some jitter before LinkPhysicalAddress calls, and I would expect a nice speedup.

Thanks! Will be trying out us-east server soon. We've tried removing the proxy and the performance is similar so we ruled that out.