Hi, during a import operation, for how long can we query the "import_status" of the import "id" ?, having "not found" issues with sleep(0.5), just decreased it to sleep (0.1), but trying to guess in case we wanted to do backoff&retry mechanisms

Hi Florentino, The duration for which you can query the

import_status

of an import ID depends on several factors, including the size of the import, system load, and the specifics of the lakeFS setup. There isn't a one-size-fits-all answer, as it can vary widely. However, import operations are typically logged and tracked within the system for an extended period, so you should be able to query the status for as long as the import task remains in the system's active or recent history. Instead of using a fixed sleep interval or a very short sleep duration like

sleep(0.1)

, consider implementing an exponential backoff strategy. Start with a longer sleep interval (e.g., 0.5 seconds) and double the interval with each retry to a certain maximum. This approach helps to balance the load on the system and increases the likelihood of a successful status query as the operation progresses. For example: • Initial sleep: 0.5 seconds • If "not found" or other retryable error occurs, double the sleep interval: 1 second, 2 seconds, up to a maximum (e.g., 4 seconds) I would also define a maximum number of retries to avoid endless loops.

so it could be a not found because I'm querying it too early? kk, doing that thanks (I already had the loop, but it was failing on any exception) actually makes sense, my sleep is after the first try

Exactly

@Florentino Sainz the status update interval is 1 second so doing anything less will be wasteful

for the early queries it can help though, unless our clocks are synced it can be a 2seconds delay (your second + my second), but will make sure after 5 tries we never go under 1s, ty