P0-4: aprender builds on ANY of intel/gx10/yoga — a shared `build` pool label, host-layout parametrized, yoga joins the x86 pool first, gx10 after an arm64 sovereign-ci image (operator 2026-09-10)

[noahgift]

Decision (operator, 2026-09-10, verbatim)

why wouldn't we let ONLY aprender use gx10 for build server if it needs it, and ditto yoga. if no issues, lets change our build process so aprender can use ANY of those three build servers, but we can always redirect builds if we need to run something special on gx10, cancel jobs, etc

Measured today

• The merge queue sat 2+ h at position 1 (fix(ptx): emitter was non-deterministic — the cubin cache has NEVER hit (42,880 stale cubins / 498 MB) #3066) with 26 aprender runs queued and zero executing: the 16 intel clean-room runners are one 32-core box shared by every paiml repo (load 102/32, 15/16 slots on bashrs/rmedia/forjar jobs). gx10 (20 cores, 119 GB, 301 GB free) and yoga (22 cores, 30 GB, 831 GB free) sat at load 0.0.
• Runner groups: gpu-nodes (gx10) is already aprender-only; gpu-x86 (yoga) is aprender + infra.
• GitHub matches ALL labels of runs-on (no OR), so "any of the three" = one shared pool label. Done live: build on the 16 intel runners (org runners API), yoga-eph and a new build-only replica gx10-build (infra PMAT-1096-ephemeral-gpu-runners, forjar-applied). scripts/check_runner_labels.sh accepts build as a pool (branch PMAT-1098-build-pool-any-of-three).
• What blocks the label switch: ci.yml is welded to the intel host: 33 hardcoded paths (/mnt/nvme-raid0/targets ×18, /mnt/nvme-raid0/cargo-ci ×12, /home/noah/data/sccache ×3) and the image localhost:5000/sovereign-ci:stable (amd64 only, 512 MB, registry on mac-server 192.168.1.149:5000 — LAN-reachable). workspace-test, guard-cargo, guard-tree, mutants all run INSIDE that image via docker run with those bind mounts. Re-labelling alone would land a job on yoga with no image and no such directories.

Plan (rows; each a PR with its own acceptance)

1. BP-1 parametrize the host layout (aprender, default-preserving): every run: block derives CI_TARGETS_ROOT, CI_CARGO_ROOT, SCCACHE_HOST_DIR, CI_IMAGE from the runner's environment with today's intel values as defaults (${CI_IMAGE:-localhost:5000/sovereign-ci:stable} …). Acceptance: diff of the rendered commands on intel is empty; a case table proves an override flows into the -v mounts and the image ref.
2. BP-2 yoga joins the x86 pool (infra): /etc/docker/daemon.json insecure-registries: ["192.168.1.149:5000"]; the disposable runner writes .env (CI_IMAGE=192.168.1.149:5000/sovereign-ci:stable, CI_TARGETS_ROOT=~/eph-work/ci/targets, CI_CARGO_ROOT=~/eph-work/ci/cargo, SCCACHE_HOST_DIR=~/eph-work/sccache); the sovereign-ci image pre-pulled. Acceptance: workspace-test dispatched on yoga green at the measured duration; RAM headroom recorded (30 GB box).
3. BP-3 flip the labels: workspace-test (then guard-cargo, guard-tree, mutants) → [self-hosted, X64, Linux, build]. Acceptance: one merge-queue run lands on yoga while intel is saturated.
4. BP-4 gx10 (aarch64): build sovereign-ci for arm64 (buildx on gx10), publish to the same registry as a multi-arch manifest; drop X64 from the pool jobs after CI's exact nextest line is green on gx10 (measurement running today). Acceptance: a queue run on gx10-build green.
5. BP-5 redirect/cancel runbook + guard: machines/clean-room/runner/build-pool.sh <box> on|off toggles the label (API for persistent runners, forjar params for disposable ones); a guard refuses a job selector only gx10 can satisfy unless it names a GPU label.

Until BP-3 lands nothing changes for CI; BP-1 is safe to merge at any time.

[noahgift]

Measured (2026-09-10 15:42–15:47Z), CI's exact nextest line (cargo nextest run --profile ci --workspace --lib --exclude aprender-gpu --exclude aprender-cuda-edge --exclude aprender-compute) inside the disposable-runner image sovereign-gpu-runner:2.337.0 (rustc 1.93.0, nextest 0.9.143 prebuilt), fresh clone of main, cold target:

box	cores / RAM	compile (cold)	suite	result
gx10 (aarch64, GB10)	20 / 119 GB	~8 min	169.6 s for 27,080 of 82,078 tests (fail-fast stopped it)	27,079 passed, 1 failed, 130 skipped; peak RAM 4 GB
yoga (x86_64, Core Ultra 9 185H)	22 / 30 GB	~8 min	134.7 s for 27,121 of 82,200	27,120 passed, 1 failed, 131 skipped; peak RAM 3 GB
intel clean-room (today's job)	shared 32-core box	—	—	73 min for the same job when the fleet is free

The single failure on both is aprender-orchestrate … test_find_apr_binary, which asserts an apr on $PATH unless CI is set — GitHub Actions sets CI=true, my containers did not; not a pool blocker. cargo test -p aprender-compute --lib (the SIMD crate) passed on both (gx10 3,325, yoga 3,510). A --no-fail-fast rerun with a persistent target dir is running on both to surface every other failure at once.

Landed so far: build label on the 16 intel runners + yoga-eph + new gx10-build (infra PMAT-1096-ephemeral-gpu-runners: 04e597d, c84a2b9 bakes nextest into the image, e821786 = BP-2 for yoga, not yet applied); BP-1 (host layout parametrized, byte-identical on intel — residual diff 0 lines) on branch PMAT-1098-build-pool-any-of-three, make gate running.