Hello, so starting a new thread here. I ran another test, and it failed in the same spot in the bulk delete part of spark. This time, I up'd the size of the lakefs server to 4 vCpu's and 8GB of ram. The lakefs server looked really good this time. CPU usage maxed out at 43% (as opposed to pegged at 100% on all previous runs). The postgres db read/write latency were both under 5ms. The only thing i can think of to try at this point is to turn on that hadoop failure retry setting to try can recover from these bulk delete failures. I'd hate to do that though because that is potentially hiding the real problem.

I understand your reluctance to allow Spark jobs to succeed when they failed to clean up their mess. Until we resolve the issue with slowness of multiple uncommitted deletions, I am afraid we will need to suggest workarounds. I will however point out that Spark can generate lots of garbage on its object store; a production system does need to collect this garbage or to live with it. Here is one instance, a question on Microsoft Azure.

Hi, so i'm going to give it another go setting the mapreduce.fileoutputcommitter.failures.attempts to 100.

Hi Joe, Thanks for sending that data! Today is a holiday here, and I'll go over them tomorrow morning.

@Joe M, looking at the logs it seems like there are a lot of errors like this one:

Caused by: io.lakefs.hadoop.shade.sdk.ApiException: Message: Unauthorized
HTTP response code: 401
HTTP response body: {"message":"error authenticating request"}

assuming you didn’t change anything with lakeFS IAM or the job creds, this is pretty odd. Can you share the lakeFS server logs during the run?

yeah i didn't not change anything. I'm the code is just sitting there running the spark partition command (and another thread is in a loop, committing periodically)

Can you share the server configuration too?

000000.gz,config.yaml

Few observations: 1. It seems like lakeFS is receiving a signal to exit from the controller. Do you know why ECS stopped lakeFS? health checks?

2024-06-11T17:52:03.210Z received SIGTERM
2024-06-11T17:52:03.210Z sending SIGINT to LakeFS process: 68
2024-06-11T17:52:03.210Z waiting for lakefs to shutdown
2024-06-11T17:52:03.210Z Shutting down...
2024-06-11T17:52:03.222Z lakefs still running, waiting...

2. With the data we see I lean towards the number of connections to the DBs. If you want to experiment with non-HA setup, I think you should increase the number of open connections to postgres which defaults to 25. What’s the max amount your postgres can handle? I would try at least 100 and see where it takes you.

after i terminated the emr cluster that was doing the partitioning (because the box became unresponsive), i shut down the lakefs server ecs task. so that would have been after everything failed and became unresponsive.

It would be something like:

listen_address: "0.0.0.0:8000"

database:
  type: "postgres"
  postgres:
    max_open_connections: 100

logging:
  format: text
  level: INFO
  output: "-"

blockstore:
  type: s3
  s3:
    region: __REGION__

from the log: database.postgres.connection_max_lifetime=3h0m0s

I guess changing

connection_max_lifetime

is worth a shot too

yeah, i'm thinking any millisecond we can save by not having to reconnect is better. since the majority of this operation happens over 2 hours, i'd rather not have the connections reopened.