Skip to content

kaizen: intel per-run cargo target dirs (263 GB) push the host over 90% and drop clean-room runners — free each run's own dir at job end #5397

Open

Description

@noahgift

Loss

Per-run cargo target dirs on the intel host hold 263 GB (/mnt/nvme-raid0/targets/aprender-ci, 27 PR trees, all < 12 h old; measured 11:3xZ) plus 145 GB in cargo-ci. The host sits at 89-92 % used, and ≥ 90 % drops its clean-room runners (C338), which is queue-wait supply (#5394).

Five whys

  1. Why do runners drop? Disk ≥ 90 % on intel.
  2. Why is the disk full? Every PR run keeps its own aprender-ci/<PR>/run-<id> target dir; 27 PR trees are live at once.
  3. Why so big? Shard 1 of the workspace tests builds the whole tree: 5385-s1 39 G, 5390-s1 25 G, 5393-s1 23 G, against 2.7-9.5 G for the other shards. This is the same shard 1 that is the slowest shard in the kaizen timing loss.
  4. Why are finished runs' dirs still there? Reclaim (scripts/ci_reclaim_target_dirs.sh) runs inside a later run of the same PR and spares live siblings; a PR with no later run keeps its dirs. Not verified: whether any age sweep runs when no new run comes.
  5. Why no end-of-job cleanup? The per-run dir (fix(ci)!: per-run target-dir — P0 STOP THE LINE on cancel-corrupt-state race #1693) was added to stop siblings deleting each other; that fixed correctness and left no step that frees the run's own dir when the job ends.

Countermeasure

Add an if: always() final step to the shard and x86-main jobs that deletes only that job's own run-<id> dir (never the PR parent), plus an age sweep on the host for dirs > 3 h idle. sccache is already warm per ci.yml, so removing the target dir does not cost the compile cache. Choose per-run cleanup over a shared target dir: a shared dir brings back the sibling-deletion race the reclaim script documents.

Owner

aprender-a3 (ci.yml change, C314 as usual). Host sweep side: infra-38.

Done when

intel / stays under 85 % for 24 h with 10+ PRs running, and aprender-ci holds no dir of a finished run older than 3 h.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High priority

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions