daemon: runtime DB sidecar divergence (deleted-open -wal) refuses all connections until restart; Web GUI hot-retries failed agent loads at ~209 rps with no backoff

[jolestar]

Summary

Two independent defects composed into a full Web GUI outage on 2026-09-13 (observed by the long-running GUI observer):

1. daemon: a runtime DB "sidecar divergence" (deleted-open -wal) makes the connection guard refuse every new connection, so all API endpoints fast-fail 500 until the daemon is restarted — a transient sidecar state escalates into a permanent self-locked outage with no recovery path.
2. web-gui: while the daemon returned 500, the frontend hot-retried the failed agent detail load with zero backoff at ~209 req/s for 21 minutes (263,356 requests to a single endpoint from one page), amplifying the outage into a self-DDoS.

Timeline (UTC, 2026-09-13)

Time	Event
05:00:02	First ERROR holon::runtime_db::connection: runtime db sidecar divergence error=runtime db sidecar divergence detected (deleted-open sidecar); refusing a new connection: db=/home/jolestar/.holon/state/runtime.sqlite, sidecar=-wal, fd=11, open_dev=66311, open_inode=89522540, canonical_identity=None (journal, daemon PID 2427897)
05:00:03 → 05:21:10	Every API request needing a DB connection fast-fails 500 (durationMs=1). Observer recorded 263,378 × 500 + 113 × 503 (~209 rps sustained), dominated by GET /api/agents/holon-growth/skills (263,356)
05:21:10	Operator stopped the daemon (systemd: "Consumed 9h 32min 48.144s CPU time" over a 1h32m uptime ≈ 6.2 cores average)
05:30:30	Daemon restarted; /api/agents/holon-growth/skills returns 200 OK immediately

Context: the daemon was already under heavy lock pressure from #2948 (memory indexer rebuild monopolization; database is locked warnings ongoing, ~200% CPU) in the same window.

Defect A (daemon): deleted-open sidecar → total connection refusal, no self-recovery

• Guard: src/runtime_db/connection.rs:418 (refusing a new connection ... preserve the files and FD/inode evidence, then perform offline recovery or restart).
• Once the guard trips, all new connections are refused — every endpoint 500s — until a manual restart. The error itself suggests the intended recovery is "offline recovery or restart", i.e. the running daemon cannot recover on its own.
• Open question: what deleted/recreated the -wal while an fd still held it open (canonical_identity=None suggests the open handle predates identity tracking). Candidate: WAL truncation/checkpoint churn under the Daemon memory indexer: one agent's inline full rebuild monopolizes the serial round loop — all cursors freeze and lag grows unbounded (#2940 follow-up) #2948 lock convoy.

Defect B (web-gui): zero-backoff hot retry loop on failed agent loads

• During the 500 window one agent page issued 263,356 requests to /api/agents/{id}/skills in 21 minutes (~209 rps, no sign of any backoff), each also emitting a console.error ("Failed to load resource: the server responded with a status of 500").
• Fetch path: client.ts getSkillCatalog(agentId) (web-gui/app/src/runtime/client.ts:995), agent-scoped loads in web-gui/app/src/runtime/runtime-store.ts:2695/2738/2783 (error stored into agentSkillCatalogErrorByAgentId).
• Whatever the driving loop is (render-triggered refetch or catch-retry), a failing fetch must back off (exponential + jitter, capped) instead of hot-looping. The browser-side amplification contributed most of the 6-core CPU burn during the outage.

Evidence

• Observer JSONL: events-2026-09-13.jsonl (527,636 lines; 263,491 http_error + 263,491 console, 1:1 paired).
• journalctl --user -u holon.service excerpts 05:00:02Z–05:21Z.
• Post-restart curl: GET /api/agents/holon-growth/skills → 200.

Fix directions

• A: root-cause the deleted-open -wal (likely interplay with heavy WAL/checkpoint churn); consider a safe in-process recovery path (re-resolve canonical identity / reopen) for this divergence class, or at minimum degrade to a bounded refusal with explicit liveness signal instead of a silent permanent 500 wall.
• B: add bounded exponential backoff (with jitter) for agent detail / skills loads and audit other fetch loops for the same pattern.

Related: #2948 (indexer lock convoy active in the same window), #2940 (prior indexer pacing fix, partial).

[jolestar]

补充 Defect A 的根因归因与修复进展（2026-09-13）：

• 直接触发器已确认：05:00:02 UTC，普通 read-write sqlite3 连接打开 live runtime.sqlite、只执行查询并关闭；其 WAL close cleanup 取得主库 EXCLUSIVE lock 后 unlink 了 runtime.sqlite-wal / -shm。
• 不是 Daemon memory indexer: one agent's inline full rebuild monopolizes the serial round loop — all cursors freeze and lag grows unbounded (#2940 follow-up) #2948 checkpoint churn 的因果结果：memory.v2.sqlite3 的锁竞争与本事故并发发生，但属于独立数据库和独立故障链。
• Holon 的缺陷边界：SQLite Unix VFS 的传统 POSIX byte-range lock 是进程关联锁；Linux 上同进程关闭任意一个指向同一主库的普通 fd，都可能释放该进程在文件上的全部 traditional locks。只依赖 SQLite writer connection 的 shared lock，无法形成足够稳健的进程生命周期不变量。fix: preserve SQLite locks during sidecar inspection #2917 的 O_PATH 修复消除了一个已知 inspection 路径，但不能把 correctness 建立在“未来代码永不普通 open/close 主库 fd”上。
• 当前审计结果：现有生产代码未发现额外对 canonical runtime.sqlite 的普通 File::open；故障回归测试已确定性复现“同进程普通 fd close → 外部 read-write observer close-cleanup 删除 sidecar”。
• 正在实施的防御性修复：Linux 上在 SQLite shared-byte range 持有进程生命周期 OFD read lock；普通外部 SQLite 进程仍可 open/query/close，正常 writer/checkpoint 继续工作，但外部 close-cleanup 无法取得覆盖该范围的 EXCLUSIVE lock。OFD lock 不会因同进程其他 fd close 而释放。
• 故障语义：保护激活失败则启动失败；激活后若仍检测到 deleted-open、canonical missing 或 inode mismatch，runtime 永久进入 quarantined，拒绝新连接和写事务，readiness/status 显式降级，DB-dependent HTTP 返回 typed 503（runtime_db_quarantined，Retry-After: 1）。首版不自动 checkpoint/reopen/reconcile，以保留未知 split-brain 现场。

当前 Linux 回归已覆盖：同进程普通 fd close、外部默认 read-write observer 反复关闭、observer 存活期间 writer 继续提交、外部 EXCLUSIVE 获取失败，以及 divergence 后不自动重建 sidecar。PR 完成全量验证后会关联本 issue。

[jolestar]

Closing: Defect A fixed and verified; Defect B split out

Defect A (daemon runtime DB sidecar divergence) — fixed:

• Root cause attributed in the comment above (2026-09-13): a plain read-write sqlite3 connection's WAL close-cleanup unlinked the live runtime.sqlite-wal/-shm sidecars. On Linux, POSIX byte-range locks are process-associated, so closing any fd on the same database can drop the whole process's traditional locks — relying solely on the SQLite writer connection's shared lock is not a robust process-lifetime invariant, and fix: preserve SQLite locks during sidecar inspection #2917's O_PATH probe did not cover the "never plain-open/close the main DB fd" invariant.
• Defensive fix: fix: protect runtime database WAL sidecars #2958 (fix: protect runtime database WAL sidecars), merged 2026-09-13 12:54 UTC (885367b) and deployed; the production daemon currently runs 91cb5560 which contains it.
• Verification: the long-running GUI observer has recorded 0 new divergence/quarantine events across continuous 2-hour observation windows since deployment (2026-09-13 → 2026-09-15), including daemon restart and page-reload activity.

Defect B (web-gui zero-backoff hot retry, ~209 rps) — not forgotten: split into #2991 so each defect keeps its own lifecycle. This issue's title referenced both defects; the web-gui half now lives there.

Resolving as fixed for Defect A.