runner 0.1.211: supervised slot processes never register — supervisors log 'supervising' then go silent (rolled back on sentry)

[donbeave]

Incident (2026-08-25 ~17:57–18:15 UTC, sentry, all 9 instances)

After the 0.1.210 → 0.1.211 apt upgrade every daemon instance came up,
logged exactly one line, and then produced no output and no GitHub
registrations:

[daemon pid=1672781] supervising 9 slot process(es) M=8 surge=1 version=0.1.211

• 50 /usr/bin/velnor-runner slot --state-dir … processes were alive but
  inert: zero JIT registration attempts, zero broker polls, zero errors in
  daemon.log, lifecycle.log or broker.log for 12+ minutes.
• Slot log files under
  /var/lib/velnor-tailrocks/runner/daemons/velnor-tailrocks/slots/*/logs/
  stopped advancing at the pre-upgrade timestamp (last write 17:50).
• Estate-wide effect: all velnor-lane CI queued (actions/runs?status=queued
  non-empty across repos).

Suspect range (v0.1.210..v0.1.211): #377 supervised-processes split is the
only behavioral change (3d1ffd3), plus 83a64e9 warm-target fingerprints.
PR #383 ('keep Ready until real GitHub assignment') was already open against
this flow before the incident.

Recovery (worked)

1. Mask template velnor-daemon@.service, stop instances.
2. velnor-runner release rollback → active record v0.1.210.
3. Also downgrade the package — verify-installed correctly refuses to
   start a 0.1.211 binary against a v0.1.210 record:
   apt-get install --allow-downgrades velnor-runner=0.1.210.
4. Unmask + start → slots ready via broker within one poll cycle; queue
   drained.

Requested

• Root-cause why supervised slot processes start but never reach the
  registration/poll loop on 0.1.211 (no error surfaced anywhere — add a
  startup-progress log line per slot process so silence is diagnosable).
• Fix forward, ship as 0.1.212+, and re-run this incident's exact upgrade
  path on sentry before declaring done.
• Add a regression gate: an upgrade smoke that asserts at least one slot
  reaches 'ready via broker' within N seconds of daemon start.

[donbeave]

Deterministic reproduction confirmed: second clean 0.1.211 deploy (daemon pid 1916699, 18:27:30 UTC) → identical silence, zero runners in the org registry, queued CI grew. Not environmental. Rolled back again (record + deb downgrade to 0.1.210): 13 broker-ready registrations within one poll cycle. Evidence paths in the main report. #383 touches this readiness path — please fold the startup-silence diagnosis in.

[donbeave]

⚠️ HOST STATE — DO NOT RE-UPGRADE TO 0.1.211

Third identical failure at 18:40 UTC (daemon pid 2004820: supervising 9 slot process(es) → silence, zero org runners). Sentry is now rolled back to the v0.1.210 record + 0.1.210 deb and must stay there until a fixed release exists.

Fix-forward procedure:

1. Diagnose why supervised slot processes never reach registration on 0.1.211 (see evidence above).
2. Land the fix + ship 0.1.212 through the normal release train.
3. Only then upgrade sentry: apt install 0.1.212 → activate → assert ≥1 'ready via broker' within 120s before declaring success.

Repeated in-place retries of 0.1.211 each starve all estate CI for ~10 minutes and have failed three times identically.