Daemon memory indexer follow-up (#2954): bounded rebuild slices never get a second round-trip — 36 open jobs all frozen at one 500-doc slice, cursors still stuck

[jolestar]

Summary

#2954 made a full memory-index rebuild resumable by capping each call at one slice
(MEMORY_INDEX_REBUILD_DEFAULT_SLICE = 500), but the next slice only runs on the next
daemon indexer round, and a round now spans hours. Result: bounded rebuild jobs start and
never finish, and because outbox consumption is skipped while a job is open, agent cursors
stay frozen exactly as before #2954.

Observed on a live daemon running 885367b9 (v0.39.0, includes #2954 + #2958), started
2026-09-13T13:26:36Z.

Evidence

1. Every open rebuild job is stuck at exactly one slice

~/.holon/.holon/indexes/memory.v2.sqlite3, read-only at 17:00Z:

select count(*), max(documents_processed), sum(documents_processed > 500),
       sum(phase = 'finalize') from memory_index_rebuild_jobs;
→ 36 | 500 | 0 | 0
min(started_at) = 2026-09-13T13:27:51Z   max(started_at) = 2026-09-13T16:58:29Z

All 36 jobs are phase = scan, source_kind_index = 1, documents_processed = 500, with
last_progress_at ≈ started_at + 1s. A new job starts every ~4–5 minutes; not one job has
ever received a second slice in 3.5 hours.

2. A round is hours long, so a slice is scheduled hours apart

• ~/.holon/agents: 630 agent directories, 573 of them tmp_child_* (E2E leftovers).
• runtime_index_outbox: 63 distinct agent ids, 44 are tmp_child_*.
• These ephemeral agents are enumerated into the work set every round
  (src/host.rs:1468-1505: identity records ∪ outbox watermark agents ∪ pending sources),
  and each visit is expensive: 36 agent refresh exceeded watchdog threshold (>30 s)
  warnings in this window.

So one serialized round over the work set spans hours; the per-agent rebuild cadence is
"one 500-document slice per round", which is slower than the producers.

3. An unfinished job freezes the cursor, not just indexing freshness

refresh_memory_index_bounded (src/memory/index.rs:376-383) skips normal outbox
consumption whenever a rebuild job exists:

let handled_rebuild = index.rebuild_job(&agent_id)?.is_some()
    || !index.rebuild_intents_for_agent(&agent_id)?.is_empty();
index.advance_rebuild(storage, batch_limit.max(1))?;
if !handled_rebuild {
    refresh_memory_index_for_storage(...)?;   // never runs while the job is open
}

Live index_status for holon-web at 16:55Z: cursor still 947440 (unchanged since
2026-09-12T16:55:57Z — 24 h), high_watermark 970644, lag 19654 → 23204 and growing,
pending_count 1167, stale_reasons includes rebuild_in_progress,
results_may_be_incomplete: true.

4. Collateral cost

database is locked 4187 times in 3.5 h (~0.33/s) while the daemon averages 240 % CPU;
in the same window the front-end read path degraded (221 roster.snapshot spans up to
16.1 s, 243 events.backfill spans totalling 1956 s up to 63.9 s, 24 × 503 on
snapshot/projection-snapshot). Note also did_work |= status.rebuild_phase.is_some()
(src/host.rs:1562-1564) keeps rounds back-to-back while any job is open, so the loop burns
CPU without making progress — the same spin shape as #2939.

Candidate directions (needs design decision)

1. Give the indexer round a time budget plus a round-robin continuation position, so a
   round cannot span hours and every agent gets a slice per round.
2. Enumerate open rebuild jobs explicitly and resume existing jobs before starting new
   ones (observed: 36 started, 0 resumed).
3. Keep leaked ephemeral agents (tmp_child_*) out of the work set — and/or clean them up at
   E2E teardown; 573 directories and 44 outbox agents currently consume round budget forever.
4. Reconsider the handled_rebuild short-circuit so a long rebuild bounds, rather than
   freezes, cursor advancement.

[jolestar]

部署后 6h 跟踪证据（2026-09-13T14:45Z–20:55Z，daemon 885367b9 含本 issue 所指 #2954 分片逻辑）

来自常驻 Web GUI 观察（周期性采集 + journald 直查），本 issue 描述的「只拿到一个 slice、后续轮不再认领」持续未解，且锁放大已扩散到消费侧：

1. holon-web 的 scan rebuild 冻结在一个 slice 上 6h10m

• rebuild_phase=scan、rebuild_started_at=14:45:34Z、rebuild_documents_processed=500、rebuild_last_progress_at=14:45:37Z——至 20:55Z 零推进（= 恰好 1 个 MEMORY_INDEX_REBUILD_DEFAULT_SLICE，与 issue 主体一致）。
• cursor=947440 冻结 28h；lag 19654→26664↑；pending_count 1135→1258↑；last_indexed_at 仍停在 09-12T16:55Z。

2. 新信号：outbox apply 失败从单点 HOL 扩散为全 agent 反复失败

18:55–20:55Z 两小时窗内 memory index outbox apply failed; retaining rows for ordered retry 共 102 次，覆盖 24 个 agent（tuptup-dev 20、holon-web 20、holon-ops/dev2/dev 各 14、其余多为 8–10），错误全部为 Error code 5: The database file is locked。即消费侧每个 agent 的 apply 都在锁竞争中被反复拒绝——lock convoy 已从「rebuild 独占」放大为「生产+消费全链路受阻」。

3. watchdog 与 tmp_child 工作集持续增长

• 同窗 agent refresh exceeded watchdog threshold (>30s) 88 次。
• started bounded memory index rebuild agent_id=tmp_child_* 约 每 4 分钟一个新 agent id（28 分钟内 8 个，suffix 单调递增），每个都 >30s watchdog——E2E 残留侧仍在为工作集注入新条目（agents 目录侧数量稳定 ~580，说明清理在工作集之外进行，但 outbox/index 侧债务持续累积）。
• database is locked 总日志速率 ~0.1/s（745 次/2h），与前窗持平。

4. 前端可感知面（同窗，供优先级参考）

• 观察器 827 事件：0 网络失败 / 0 HTTP 错误；events.backfill 18 次且 18/18 incomplete（cursor 冻结的直接前端映射，单次最长 12.5s）。
• 慢 API 中 projection-snapshot 3,985 字节响应构建 9.4s、holon-dev/state 113KB 响应 6.3s——纯锁等待，非数据规模问题。

验收提醒（部署修复后复测口径）

SELECT count(*) FROM <rebuild jobs> WHERE documents_processed > 500 > 0、holon-web cursor 恢复推进、lag 回落、per-agent apply failed 速率归零、/api/agents/*/state 首访 p95 回到亚秒。

（观测数据源：~/.holon/agents/holon-web/work/web-observer/events-2026-09-13.jsonl + journalctl --user -u holon.service，可按需补充采样。）

[jolestar]

部署后清偿验证（PR #2968 链，daemon 本地构建 47e58397，2026-09-14T04:17Z 起）

本 issue 立案时的全部核心病理已实测解除。证据来自常驻 Web GUI 观察（周期采样 + journald 直查 + runtime.index_status）：

1. Cursor 恢复推进，lag 收敛至基线（Evidence §3）

时点 (UTC)	cursor	lag	last_indexed_at
09-13T14:45（立案）	947,440 冻结	23,204 ↑	09-12T16:55（冻结 24h+）
09-14T02:55	951,401	28,713 → 24,091 ↓	解冻，秒级刷新
09-14T04:55	959,965	24,091 ↓	实时
09-14T06:55	987,295	0	2026-09-14T06:55:48Z（实时）

stale_reasons 从 [dirty_marker, backfill_incomplete, outbox_lag] 收敛为 [pending_sources, rebuild_in_progress]（后者为进行中的 scan 正常状态）。

2. Rebuild 跨片推进，不再「一 job 一 slice 饿死」（Evidence §1/§2）

• 立案形态：36 个 open job 全部恰好停在 MEMORY_INDEX_REBUILD_DEFAULT_SLICE = 500 文档、零个拿到第二片。
• 当前：scan 于 05:58:31Z 启动，06:55Z 已推进 rebuild_documents_processed = 4,440（≈8.9 片），rebuild_last_progress_at = 06:55:29Z 与 last_indexed_at 同步秒级刷新——分片在跨轮持续续片。

3. 附带代价消退（Evidence §4）

• database is locked：0 次 / 2h38m（立案时 4,187 次 / 3.5h ≈ 0.33/s；上一部署链曾达 1.35/s）。
• daemon 稳态 RSS 707MB、CPU ~214%（此前同负载下 8–10GB + 240–270% 且无进展）。
• 前端读路径 2h 窗：0 次 503 风暴（30 次 snapshot 503 全部 1–4 次/分散分钟、高负载窗，为已知超时家族）、467 采样 heap 257–391MB maxStepUp = 0、无 #2905 形态回填风暴（101 次 backfill ≤1,000 事件/片、最长 15s，主体 holon-reviewer 22 次，属正常追赶密度）。

仍开放的关联项

• Candidate direction 3（tmp_child_* 泄漏 agent 仍会进入工作集并占用 rebuild 槽位）尚未处理——目前 573 个残留目录。债务清偿达成说明有界轮转能排干，但清理侧仍值得单独跟进。
• 新一轮 scan（05:58Z 启动）仍在推进中，本轮观察会继续验证其完成情况。

部署状态说明

PR #2968 已合并进 main（merge commit 1209eb5a，2026-09-14T05:07:20Z）。上述验证运行在包含该修复链的本地构建 47e58397（04:17Z 部署）上；远端发布版本覆盖该修复后即可视为完全闭环，故暂不关闭本 issue，等待 scan 完成 + 发布确认。

—— 常驻观察：work_8f6806e922eb809（数据窗 04:54–06:55Z，事件 1,698 条 + journald + index_status 直查）

[jolestar]

已确认本 issue 的核心故障已由 PR #2968 修复：rebuild 不再因昂贵的 FTS 全表扫描而长期停留在第一片，daemon 已采用更小的 rebuild batch 并在有效 slice 后主动让出执行权。2026-09-15 对 v0.40.0 的生产只读核验显示，现有 rebuild jobs 已能跨片推进，holon-web rebuild 已完成，cursor/watermark 对齐且 pending=0。\n\ntmp_child_* 生命周期残留、rebuild 期间 outbox 暂停以及更强的公平性指标属于可独立跟踪的后续债务，不再阻塞本 issue 关闭。