`restart_workers` can kill already-started jobs

[bennoschoenstein]

What?

With restart_workers=True, QUEENS restarts a worker after each of its jobs finishes. But the worker does not wait for that restart: it immediately starts its next job. So the restart always hits a job that is already running.

To soften this, every job first sleeps 5 s before doing real work (_dask.py, lines 101–106). The idea: the restart arrives within those 5 s and only kills a sleeping job, which is cheap to redo elsewhere.

With many short jobs, the restart often arrives too late. The restart signal is late because QUEENS sends the restarts one after another, and with many short jobs, many finish at the same time.

Consequences

Killed jobs redo their work from scratch. Each kill counts as a failure; enough of them abort the whole run (KilledWorker), although no job actually failed. The 5s sleep alone costs a lot of core-hours potentially (when there are a lof of short running jobs).

Long jobs hide the problem: restarts arrive fast enough, and 5 s do not matter. #57 is similar.

Possible fixes, from small to large

1. Document it: keep False for short jobs; True costs 5 s per job and kills the next job.
2. Make the 5 s delay configurable.
3. Let workers restart on their own between jobs (e.g., Dask's lifetime option) instead of being restarted from outside. This would remove both the sleep and the kills, and probably fix Not all workers are restarted on cluster #57 too.