So, good news/bad news on setting fs.lakefs.delete.bulk_size to 50. It did seem to fix the bulk delete problem. but, now it's failing after that in a different place. Also there's a few exceptions at the beginning with this message, not sure what it means

Caused by: java.io.IOException: Failed to finish writing to presigned link. No ETag found.

I think the operation was retried and it succeeded cuz my job didn't fail on those. It only failed on the timeouts.

That’s good to hear @Joe M. I don’t see anything from the lakeFS logs and the Spark logs are all over the place. Can you share the exact stacktrace of the Spark failure? Also - did you manage to run more than a single lakeFS server?

lakefs server logs

Hey Joe, running multiple servers is essential for any HA setup. Moreover, not sure I fully understand the value of doing a stress test poc on a single server. Is running a single server a requirement for some reason?

Right, this is the question i keep asking. I'm not sure what is considered a stress test for this product. Like, which part of what we're doing are you saying is stressing lakefs? Is it the number of partitions? the amount of data? both? or something else?

I said it’s the amount of objects written together with the fact that it’s a single server. Considering the fact that we estimated roughly a 20M API calls during 2-3 hours, I’d expect several minutes of >10k req/sec to that server. That’s a lot, while I’ve seen many installations that handle bigger bursts, they were never relying on a single server for that.

ah, ok. I missed that part of the discussion.