No timeout anywhere in process(): one stalled or dead worker hangs the whole run indefinitely (+5 related bugs)

[cmosig]

Summary

A long-running job (many consecutive sentle.process() calls, PlanetaryComputer, S2_cloud_classification=True, num_workers=16) froze at 542/543 ptiles and stayed there for ~19 hours until I interrupted it. Nothing timed out, nothing errored, nothing was logged.

Root cause is not one bug but a property of the current design: there is no timeout on any blocking call in the process() path. One task that stalls — for any reason — freezes the entire cube forever, and neither joblib nor sentle can notice.

Audited on the installed package (2026.7.2, identical to d03d3d5), with every claim below verified at runtime unless marked inferred:

blocking call	timeout	file
Parallel(n_jobs=..., batch_size=1)	none	sentle/sentle.py:1124
GDAL/libcurl reads on signed PC COGs	none — all seven GDAL_HTTP_* knobs are None	sentle/stac.py:75-77
StacApiIO HTTP requests	timeout=None → sock.settimeout(None)	sentle/stac.py:29
response_queue.get() (cloud mask)	none	sentle/cloud_mask.py:156
FileLock(sync_file_path)	timeout=-1 (block forever)	sentle/sentle.py:219

19 hours isn't a long timeout — it's an absent one.

---

Symptom

processing: 100%|...| 543/543 [41:35<00:00,  4.60s/ptiles]      <- cube 1 ok
processing: 100%|...| 543/543 [24:04<00:00,  2.66s/ptiles]      <- cube 2 ok
processing:  99%|...| 542/543 [1:19:33<00:09,   9.72s/ptiles]   <- cube 3, one task outstanding
processing:  99%|...| 542/543 [20:28:44<02:16, 136.02s/ptiles]  <- 19h later, unchanged
^C
  File "sentle/sentle.py", line 1124, in process
    Parallel(n_jobs=num_workers, batch_size=1)(...)
  File "joblib/parallel.py", line 1800, in _retrieve
    time.sleep(0.01)
KeyboardInterrupt

The main thread is spinning in _retrieve on a job whose status is TASK_PENDING. With timeout=None, get_status short-circuits and returns TASK_PENDING instantly, forever, while _wait_retrieval stays True because n_completed_tasks < n_dispatched_tasks.

Two things worth recording because they mislead:

• The stuck ptile is not necessarily the last timestamp. _retrieve blocks on the lowest dispatch-index incomplete job, while all later completed results sit unretrieved.
• The rising s/ptiles is an artifact, not a slowdown — it's tqdm's mean being dragged by one frozen entry.

What 542/543 proves, and what it doesn't

TqdmBatchCompletionCallback.__call__ (sentle/utils.py:23) ticks before super().__call__(), and batch_size=1 makes every batch one task, so exactly one ApplyResult._set never ran.

But that does not mean a worker was alive and stuck. multiprocessing.Pool has no dead-worker notification: _join_exited_workers reaps the corpse and _repopulate_pool_static starts a replacement, but neither touches self._cache (bpo-22393, still open in 3.12). A SIGKILLed worker and a worker in time.sleep(100000) produce byte-identical output — same frozen bar, same traceback. joblib adds nothing here (joblib/pool.py overrides only __init__, _setup_queues, terminate, _temp_folder).

So there are two live candidate mechanisms and the post-mortem cannot separate them:

1. (inferred) a worker died hard — OOM-kill or segfault in GDAL/openjpeg/torch. Circumstantial evidence on my box: an orphaned /dev/shm/psm_* block of exactly 25,719,552 bytes = 12×732×732×4, i.e. a worker_get_cloud_mask input block (cloud_mask.py:139) whose unlink() in the finally never ran. So this class of hard worker death has happened here before.
2. (mechanism proven, occurrence inferred) a worker blocked forever in libcurl — see bug 1.

Either way sentle should survive it. Suggestion: Parallel(..., timeout=...) at sentle.py:1124 converts both into a real TimeoutError (verified working on this stack, and it's a per-job head-of-queue budget, not wall-clock — 8×3 s tasks at n_jobs=2 with timeout=5 completed in 12.0 s with no false positive). One caveat: the raise propagates out of process() and skips the cleanup at sentle.py:1129-1144, leaking the cloud service and the manager — so it needs a try/finally.

---

Bug 1 — no GDAL HTTP timeouts on the PlanetaryComputer path

# sentle/stac.py:75-77
def rasterio_env(self):
    # PC hrefs are plain (signed) HTTPS -> no special GDAL config needed
    return contextlib.nullcontext()

The comment is right about correctness and wrong about liveness. Verified in the live interpreter (GDAL 3.10.3 / rasterio 1.4.4): GDAL_HTTP_TIMEOUT, GDAL_HTTP_CONNECTTIMEOUT, GDAL_HTTP_LOW_SPEED_LIMIT, GDAL_HTTP_LOW_SPEED_TIME, GDAL_HTTP_MAX_RETRY, GDAL_HTTP_RETRY_DELAY, GDAL_HTTP_TCP_KEEPALIVE are all None. libcurl's CURLOPT_TIMEOUT defaults to 0 = never.

Reproduced against purpose-built stall servers: rasterio.open / dr.read against a socket that accepts and then goes silent — both pre-header and mid-body after a correct 206 + Content-Range — blocks indefinitely with no error. GDAL_HTTP_TIMEOUT=5 turns it into a RasterioIOError in 10 s; LOW_SPEED_LIMIT=1000 + LOW_SPEED_TIME=5 in 15 s.

Exposure is large. Running the real obtain_subtiles against the shipped grid, a 30 km AOI yields 25–47 subtiles per ptile; ×12 bands ×2 acquisitions ×several range GETs ⇒ roughly 3×10⁵–1.4×10⁶ HTTPS GETs per cube. At that volume a single stall is a ~10⁻⁶ event, so clean runs either side of a hang are expected, not counter-evidence.

Suggested fix — both providers, since the CDSE Env at stac.py:131-139 is equally unprotected:

def rasterio_env(self):
    return rasterio.Env(
        GDAL_HTTP_TIMEOUT=300, GDAL_HTTP_CONNECTTIMEOUT=30,
        GDAL_HTTP_LOW_SPEED_LIMIT=1000, GDAL_HTTP_LOW_SPEED_TIME=60,
        GDAL_HTTP_MAX_RETRY=5, GDAL_HTTP_RETRY_DELAY=2,
        GDAL_HTTP_TCP_KEEPALIVE="YES",
    )

LOW_SPEED_* is the one that actually catches a stall; GDAL_HTTP_TIMEOUT alone has to be generous enough not to false-abort large range reads. Note GDAL_HTTP_MAX_RETRY on its own does nothing here: retries fire on status codes and curl errors, and a silent socket produces neither until a timeout rule converts it into one.

As a workaround users can export these as env vars today — GDAL resolves config at call time and rasterio.Env only overrides the keys it names, so it composes with the CDSE path.

Bug 2 — StacApiIO is created without a timeout

# sentle/stac.py:29
return StacApiIO(max_retries=retry)

pystac_client/stac_api_io.py:48 defaults timeout=None → :214 session.send(prepped, timeout=None) → sock.settimeout(None). Runtime-verified: get_stac_api_io().timeout is None, and ss -tnop on a blocked child shows no kernel timer at all (urllib3 sets TCP_NODELAY but never SO_KEEPALIVE).

The Retry(total=15, ...) policy does not help: urllib3 retries on exceptions, and a blocked recv() never raises one.

Much lower exposure than bug 1 — only ~3 requests per ptile — but the same unbounded failure mode. Suggested: StacApiIO(max_retries=retry, timeout=(10, 60)). Worth also setting retry_after_max on the Retry: urllib3's DEFAULT_RETRY_AFTER_MAX is 21600 s, so 16 honoured Retry-After headers is a legal 96-hour "bounded" wait.

Bug 3 — untimed response_queue.get(), and the cloud service dies silently

worker_get_cloud_mask waits with no bound:

# sentle/cloud_mask.py:156
response_queue.get()

and cloud_prediction_loop (cloud_mask.py:54-90) has no try/except, so any exception in compute_cloud_mask — CUDA OOM, a driver error, or the assert array.shape == (12, 732, 732) — kills the single service process. Every worker then waits forever.

This was not my hang (the service was alive and idle in request_queue.get() at interrupt time, and a Manager put() is a synchronous RPC, so anything the workers submitted had provably been answered), but it's a one-line fix away from being someone's:

response_queue.get(timeout=600)

plus wrapping the loop body and pushing the exception back onto request["response_queue"] instead of dying.

Bug 4 — the cleanupqueue backend has been dead code since joblib 1.4

# sentle/utils.py:51-53
def apply_async(self, func, callback=None):     # joblib <= 1.3 API
    cbs = MultiCallback(callback, self.callback)
    return super().apply_async(func, cbs)

joblib ≥ 1.4 dispatches via backend.submit(...) (parallel.py:1437). ParallelBackendBase.submit has a deprecation shim that forwards to apply_async, but PoolManagerMixin.submit (_parallel_backends.py:334) overrides it and wins the MRO — so the shim never runs and no DeprecationWarning is emitted. Runtime-verified:

>>> ImmediateResultBackend.submit.__qualname__
'PoolManagerMixin.submit'

MultiCallback is never constructed and ImmediateResultBackend.callback never runs. Independent confirmation that it can't have been running: process_ptile returns a bare int (sentle.py:229), so GLOBAL_QUEUES.pop(result[0]) would TypeError on task #1. setup.py:26 pins joblib>=1.4.2, i.e. the whole supported range.

Consequence: the per-job response queues are never released as jobs complete, so all 543 SyncManager queues stay alive for the duration of a cube, and the module-level GLOBAL_QUEUES dict is never emptied — sentle.py:1137 GLOBAL_QUEUES = dict() is a dead function-local store (process() declares only global GLOBAL_QUEUE_MANAGER; bytecode-verified: GLOBAL_QUEUES is in process.__code__.co_varnames and not in co_names). So it accumulates stale proxies across successive process() calls in a long-running process.

Fix: rename the hook and make it exception-proof, because it runs inside ApplyResult._set before _event.set() — a KeyError there orphans the job (and pool.py:594-597 swallows it), and any other exception kills the result-handler thread:

def submit(self, func, callback=None):
    return super().submit(func, MultiCallback(callback, self.callback))

def callback(self, result):
    GLOBAL_QUEUES.pop(result, None)   # result is a bare int, and must never raise

Also add global GLOBAL_QUEUES in process() (or drop line 1137).

Bug 5 — uninitialized memory can be written as reflectance

# sentle/sentinel2.py:204
subtile_array = np.empty((len(download_bands), S2_subtile_size, S2_subtile_size),
                         dtype=np.float32)

The except rasterio.errors.RasterioIOError at sentinel2.py:279 warns and continues without ever assigning subtile_array[i], so that band keeps whatever heap garbage np.empty handed back. It is then fed to the cloud classifier and written to the zarr as reflectance. The only downstream guard checks s2_crs / s2_tile_transform, which a successful B02 alone satisfies.

This matters especially in combination with bug 1: every timeout added above converts hangs into caught RasterioIOErrors, i.e. it starts exercising this path. Worth fixing in the same change — np.zeros, or better, track which band indices actually loaded and drop/mask the subtile if any failed.

Bug 6 — shutil.rmtree() on a lock file

# sentle/sentle.py:1142
shutil.rmtree(sync_file_path)   # sync_file_path is a file (sentle.py:543-544)

Always raises NotADirectoryError, always swallowed by the bare except. Result on a shared machine here: 6871 stale /tmp/sentle_*.lock files. Should be os.unlink.

---

Suggested priority

1. rasterio_env() timeouts (bug 1) + StacApiIO(timeout=...) (bug 2) — these are the unbounded-wait sources users actually hit.
2. np.empty → correct handling (bug 5) — do it together with 1, since 1 makes 5 reachable.
3. Parallel(timeout=...) with a try/finally, plus response_queue.get(timeout=...) and a guarded cloud loop (bug 3) — defence in depth for the worker-death case that no timeout inside the worker can cover.
4. submit rename (bug 4), os.unlink (bug 6).

Environment

sentle 2026.7.2, Python 3.12, Linux. joblib 1.5.3, rasterio 1.4.4 / GDAL 3.10.3, pystac-client 0.9.0, pystac 1.14.3, planetary-computer 1.0.0, zarr 3.1.5, filelock 3.20.3, torch 2.10.0, numpy 1.26.4, tqdm 4.67.1.

Call:

sentle.process(
    zarr_store=dst, target_crs=f"EPSG:{utm_epsg}", target_resolution=10,
    bound_left=..., bound_right=..., bound_bottom=..., bound_top=...,   # ~30 km block
    datetime="2015-07-01/2025-11-20",
    S2_mask_snow=True, S2_cloud_classification=True, S2_cloud_classification_device="cuda",
    S2_apply_cloud_mask=True, S2_apply_snow_mask=True, S1_assets=None,
    time_composite_freq="7d", num_workers=16,
    zarr_store_chunk_size=dict(x=250, y=250, time=10),
    processing_spatial_chunk_size=3000, save_as_uint16=True,
)

Ruled out along the way

Recorded so nobody re-treads it: the response_queue.get() deadlock (service was idle, and Manager put() is a synchronous RPC); CUDA OOM (peak reserved 300 MiB vs 19.9 GiB free, and the service was alive); FileLock contention (one outstanding task can't contend; fcntl.flock releases on SIGKILL; the lock path is unique per process() call); disk/NFS (local ext4 NVMe, 1.4 T free); /dev/shm exhaustion (378 G tmpfs, 27 M used — and SharedMemory(create=True) raises rather than blocks); a stalled task generator (reproduced: freezes tqdm below the dispatch frontier, so reaching 542 proves the generator drained); Retry backoff (ladder 0,2,4,8,16,32,64,120×9 = 1206 s per urlopen, ≤4 urlopens ⇒ ~80 min ceiling); an expired SAS token (measured TTL 86,399 s, and an expired one gives HTTP 403 → caught RasterioIOError); and a KeyError escaping the completion callback (reproduced — it hangs, but the bar reaches a full 543/543, and we saw 542).