You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
kaizen: intel per-run cargo target dirs (263 GB) push the host over 90% and drop clean-room runners — free each run's own dir at job end #5397
Per-run cargo target dirs on the intel host hold 263 GB (/mnt/nvme-raid0/targets/aprender-ci, 27 PR trees, all < 12 h old; measured 11:3xZ) plus 145 GB in cargo-ci. The host sits at 89-92 % used, and ≥ 90 % drops its clean-room runners (C338), which is queue-wait supply (#5394).
Five whys
Why do runners drop? Disk ≥ 90 % on intel.
Why is the disk full? Every PR run keeps its own aprender-ci/<PR>/run-<id> target dir; 27 PR trees are live at once.
Why so big? Shard 1 of the workspace tests builds the whole tree: 5385-s1 39 G, 5390-s1 25 G, 5393-s1 23 G, against 2.7-9.5 G for the other shards. This is the same shard 1 that is the slowest shard in the kaizen timing loss.
Why are finished runs' dirs still there? Reclaim (scripts/ci_reclaim_target_dirs.sh) runs inside a later run of the same PR and spares live siblings; a PR with no later run keeps its dirs. Not verified: whether any age sweep runs when no new run comes.
Add an if: always() final step to the shard and x86-main jobs that deletes only that job's own run-<id> dir (never the PR parent), plus an age sweep on the host for dirs > 3 h idle. sccache is already warm per ci.yml, so removing the target dir does not cost the compile cache. Choose per-run cleanup over a shared target dir: a shared dir brings back the sibling-deletion race the reclaim script documents.
Owner
aprender-a3 (ci.yml change, C314 as usual). Host sweep side: infra-38.
Done when
intel / stays under 85 % for 24 h with 10+ PRs running, and aprender-ci holds no dir of a finished run older than 3 h.
Activity