Does Lakefs-spec support async?
# helpi
Ion
04/30/2024, 2:09 PMDoes Lakefs-spec support async?
o
Oz Katz
04/30/2024, 2:10 PM
might want to ask on their github repo, not sure the maintainers monitor this channel
i
Ion
04/30/2024, 2:12 PMOk done
Ion
04/30/2024, 2:13 PMTrying to do many concurrent reads, but after some time it has so many threads it starts breaking down due to too many connections
Ion
04/30/2024, 2:18 PMHmm even with a threading.sephamore it happens 🤔
n
Nicholas Junge
04/30/2024, 2:19 PMhey. I’ll reply separately on GH as well, but we don’t support async file system operations right now.
The main reason is that we’re not sure if this is possible with OpenAPI clients like lakeFS’s (their generated code uses the multiprocessing module, and advises against async for perf reasons).
On a technical level, it might still be doable, but performance is another issue. Perhaps you can fall back to the S3 gateway for now (the S3 file system supports async)?
i
Ion
04/30/2024, 2:21 PMI see, thanks for suggestion, will try with s3fs
Ion
04/30/2024, 3:20 PMHmm I can't establish a connection with s3fs🤔
👍 1
8 Views