Matija Teršek
02/02/2023, 3:32 PMupload to backing store: MissingRegion: could not find region configuration
I believe this can be simply solved by setting ``blockstore.s3.region``, but what should we set it to when we are not using AWS directly?Elad Lachmi
02/02/2023, 3:38 PMblockstore.s3.discover_bucket_region
to falseMatija Teršek
02/02/2023, 3:40 PMElad Lachmi
02/02/2023, 3:40 PMMatija Teršek
02/02/2023, 3:41 PMLAKEFS_BLOCKSTORE_S3_DISCOVER_BUCKET_REGION=false
Elad Lachmi
02/02/2023, 3:43 PMAriel Shaqed (Scolnicov)
02/02/2023, 3:43 PMMatija Teršek
02/02/2023, 3:45 PM--direct
in command line causes the error yeah.
So for --direct mode, AWS cli is also required?Elad Lachmi
02/02/2023, 3:55 PMlakectl
, right?Matija Teršek
02/02/2023, 4:13 PMAriel Shaqed (Scolnicov)
02/02/2023, 4:19 PMBarak Amar
02/02/2023, 4:27 PM--direct
with --pre-sign
Matija Teršek
02/02/2023, 4:39 PMBarak Amar
02/02/2023, 4:39 PMAriel Shaqed (Scolnicov)
02/02/2023, 4:58 PMConor Simmons
02/02/2023, 5:36 PMThat passes data directly to s3, and only performs metadata operations on lakeFS.And the docs say this
But I'm still a bit confused on what the difference is between usingwrite directly to backing store (faster but requires more credentials)
--direct
/ --pre-sign
vs without? Why should it be faster?Elad Lachmi
02/02/2023, 5:40 PM--pre-sign
is more or less the best of both worlds
the client gets a pre-signed URL from lakeFS and uploads directly to S3--direct
, but also doesn't require the client to have access to the backing store, like a "regular" uploadConor Simmons
02/02/2023, 5:44 PMElad Lachmi
02/02/2023, 5:46 PMConor Simmons
02/02/2023, 8:07 PM--pre-sign
it seems like it still took over 2 hours to upload 2.5 GBElad Lachmi
02/02/2023, 8:09 PMMatija Teršek
02/02/2023, 8:10 PMElad Lachmi
02/02/2023, 8:36 PMConor Simmons
02/02/2023, 8:44 PMI saw 5000 on some previous conversation, but that sounds really slow so I'm making sure.to answer this one - it's 5000 images and 5000 json files, so 10,000 total
--pre-sign
, we didn't let it run all the way but the tqdm estimate looping through the files was 2-4 hours. This was without --recursive
but the test with --pre-sign
had the recursive flagElad Lachmi
02/02/2023, 8:48 PMConor Simmons
02/02/2023, 8:48 PMMatija Teršek
02/02/2023, 10:20 PMLAKEFS_DATABASE_POSTGRES_TYPE=postgres
LAKEFS_BLOCKSTORE_TYPE=s3
LAKEFS_BLOCKSTORE_S3_FORCE_PATH_STYLE=true
LAKEFS_BLOCKSTORE_S3_DISCOVER_BUCKET_REGION=false
LAKEFS_BLOCKSTORE_S3_CREDENTIALS_ACCESS_KEY_ID
LAKEFS_BLOCKSTORE_S3_CREDENTIALS_SECRET_ACCESS_KEY
LAKEFS_DATABASE_CONNECTION_STRING
LAKEFS_AUTH_ENCRYPT_SECRET_KEY
LAKEFS_DATABASE_POSTGRES_MAX_IDLE_CONNECTIONS=10
LAKEFS_DATABASE_POSTGRES_MAX_OPEN_CONNECTIONS=22
Vino
02/02/2023, 10:26 PMAriel Shaqed (Scolnicov)
02/03/2023, 6:54 AMBucket is US west, DB and server are both east.Ouch, this may be an issue: lakeFS is best used when DB, server and bucket are in the same region. It would be ideal if you could test with the bucket and the client in the same region also. If you can, you really should run bucket, DB and server close to one another. Everything from now on assumes that for some reason you cannot. If I had to guess, I would say that the issue is metadata not data. You are uploading 10_000 files IIUC, but only 2.5GiB in total. And that takes you over 2 hours. You're clearly not running out of data throughput anywhere, so I guess the cross-region metadata operations on lakeFS are slowing everything down. Where is the client located? A direct upload requires 2 API calls on lakeFS, and a long-running API call on the bucket. If you're running the client near the bucket on the west coast, each API call is at least the inter-region RTT -- this is on average 62ms (us-east-1 to us-west-1 or us-west-2), and you'll be limited to fewer than 8 uploads per second in the best case. If you need to reconnect the client, that's another round-trip. Ideally you would not be running lakeFS far from the data store. Doing it like this will be slow and expensive because of all the cross-region transfers. If all this is true, one way to speed up uploads would be to perform them massively in parallel. Since lakeFS is not busy in this case, we could easily perform many hundreds of these uploads in parallel, maybe add some jitter before LinkPhysicalAddress calls, and I would expect a nice speedup.
Matija Teršek
02/03/2023, 9:44 PM