We are seeing quite often S3 timeouts when we are doing many concurrent runs, not sure if it's because of LakeFS or rather timeouts from the underlying Azure Storage. Any suggestions or ideas?

It could be a lot of things, from network throughput to DB throttling. What’s your setup?

We have Lakefs deployed on k8s, then we use adls2 as our object store, which sits behind a vnet

@Ion 1. How many concurrent runs were you running when you received timeouts? 2. Is lakeFS deployed on Azure cloud or on-prem? 3. Are you using Postgres database or Cosmos DB? 4. What tool/language are you using to access the data in lakeFS repo? 5. Are you using S3A Gateway, pre-signed URL or lakeFS File System (lakeFSFS)? Can you please provide lakeFS logs?

1. Around 40 concurrent steps 2. Azure cloud 3. PostgreDB in Kubernetes (we are still waiting on an Azure postegresQL db) 4. Python high level API 5. S3 gateway since we use delta-rs to write data I would need to check the logs, and see what I can share

I’m going based on this previous message . Usually context cancellation at this point cause by client disconnection. Any chance you have more logs from the request context on the server side or capture an error in the client side?

Hi @Iddo Avneri, on the request side we see Generic S3 error: request or response body error: operation timed out

Hey @Stefan Verbruggen & @Ion, requests that are being cancelled during load could be the result of several root causes: 1. Your LB (Azure Application Gateway?) is cancelling the requests since its configured timeout is too low. 2. lakeFS instances running on k8s with VMs that have too low network bandwidth per your usage. Or alternatively too few lakeFS replicas. 3. Postgres requests are taking too long (unlikely for your use case) due to low connections amount, not enough resources on the DB itself, etc 4. adls2 poor performance I would explore metrics from the LB, k8s/vms, postgres, adls2 to try and narrow down the problem or just increase the resources until it’s fixed. lakeFS also exposes metrics under

<lakefs_url>/metrics

that you can fetch and analyze. This is slightly outdated, we now also have

kv_request_duration_seconds

for DB latency.

Thanks for your help on this @Itai Admi!

Thanks for the pointers Itai we will investigate this!