Hi, during a import operation, for how long can we...
# help
Hi, during a import operation, for how long can we query the "import_status" of the import "id" ?, having "not found" issues with sleep(0.5), just decreased it to sleep (0.1), but trying to guess in case we wanted to do backoff&retry mechanisms
Hi Florentino, The duration for which you can query the
of an import ID depends on several factors, including the size of the import, system load, and the specifics of the lakeFS setup. There isn't a one-size-fits-all answer, as it can vary widely. However, import operations are typically logged and tracked within the system for an extended period, so you should be able to query the status for as long as the import task remains in the system's active or recent history. Instead of using a fixed sleep interval or a very short sleep duration like
, consider implementing an exponential backoff strategy. Start with a longer sleep interval (e.g., 0.5 seconds) and double the interval with each retry to a certain maximum. This approach helps to balance the load on the system and increases the likelihood of a successful status query as the operation progresses. For example: • Initial sleep: 0.5 seconds • If "not found" or other retryable error occurs, double the sleep interval: 1 second, 2 seconds, up to a maximum (e.g., 4 seconds) I would also define a maximum number of retries to avoid endless loops.
so it could be a not found because I'm querying it too early? kk, doing that thanks (I already had the loop, but it was failing on any exception) actually makes sense, my sleep is after the first try
@Florentino Sainz the status update interval is 1 second so doing anything less will be wasteful
gratitude thank you 1
for the early queries it can help though, unless our clocks are synced it can be a 2seconds delay (your second + my second), but will make sure after 5 tries we never go under 1s, ty
🚀 2