Repository navigation
P0: nightly publishes 1 of 28 [[bin]]s (apr CPU only) and is RED today — build every bin at main head + publish a SHA manifest for lambda #4189
Description
Activity
Folding a gap in here instead of opening a new issue (measured 2026-09-24):
pvhas no nightly binary. It is one of the [[bin]]s this issue covers, and thenightlyrelease (published 2026-09-23T01:22Z) still ships onlyapr-{x86_64,aarch64}-unknown-linux-gnuand nopvasset. #4186 already requires dogfood hosts to run only the latest green nightlypv, which cannot happen until one is published.Operator rule: "nightly binary built is hard requirement for sovereign stack".
Acceptance criterion: a scheduled nightly workflow that
- builds only when
mainmoved since the last nightly and CI is green on that exact SHA; - publishes the binary with a
sha256(to thenightlyprerelease, stamped with the SHA it was built from); - when red, publishes no binary, and the red night is tracked (visible in the run and in fleet lane liveness), not silently skipped.
- builds only when
Debug run: every workspace
[[bin]],--version, at aa7c6ef (aprender-48, 2026-09-24)Built in a detached worktree at
aa7c6ef03withcargo build --bins(debug profile, default features). Then each executable ran<bin> --version. Columns: bin, rc, first line.alimentar 0 alimentar 0.69.0 apr 0 apr 0.69.0 (aa7c6ef03) apr-corpus-ingest 0 apr-corpus-ingest 0.69.0 apr-qa 0 apr-qa 0.69.0 apr-qa-readme-sync 0 apr-qa-readme-sync 0.69.0 aprender-cbtop 0 cbtop 0.69.0 aprender-cgp 0 cgp 0.69.0 aprender-compute-xtask 0 aprender-compute-xtask 0.69.0 aprender-db 0 trueno-db 0.69.0 aprender-explain 0 trueno-explain 0.69.0 aprender-orchestrate 0 batuta 0.69.0 aprender-profile 0 renacer 0.69.0 aprender-ptx-debug 0 aprender-ptx-debug 0.69.0 aprender-test-cli 0 probador 0.69.0 aprender-train-bench 0 entrenar-bench 0.69.0 aprender-train-distill 0 entrenar-distill 0.69.0 aprender-train-inspect 0 entrenar-inspect 0.69.0 aprender-train-lora 0 entrenar-lora 0.69.0 aprender-train-shell 0 entrenar-shell 0.69.0 aprender-zram-generator 0 trueno-zram-generator 0.69.0 presentar 0 presentar 0.69.0 ptop NOBIN pv 0 pv 0.69.0 (aprender provable-contracts verifier) score NOBIN simular 0 simular 0.69.0 trueno-rag 0 trueno-rag 0.69.0 trueno-zram 0 trueno-zram 0.69.0 verificar 0 verificar 0.69.0- 26 of 28 have rc 0, and every one prints the crate version
0.69.0as a whole token. ptopandscoreare NOBIN under default features: they need--features ptop,scoreon aprender-present-terminal. aprender-30's 0.70 gate: every aprender [[bin]] builds and passes a --version/--help smoke at main HEAD (per-bin RED tickets) #4194 built all 29 targets in release--locked.- Only
aprprints a SHA. So per the cop ruling, PR fix(nightly): per-arch publish, pv, CI-green gate and a SHA manifest (Refs #4189) #4206 checks rc 0 + crate version, and binds the SHA viagreen_sha+bin_sha256. A shared build-sha in every bin is Every [[bin]] --version prints the build SHA (shared build-sha), then #4189 nightly requires it #4219 (0.70.0).
Manifest (consumer contract, schema
aprender-nightly-manifest/v1): assetnightly-manifest.jsonon thenightlyrelease. Fleet local copy at${APR_NIGHTLY_MANIFEST:-$HOME/.cache/aprender/nightly-manifest.json}. Accept rule:sha256(exe) == bin_sha256, and a non-nullversion_shamust appear as a prefix in--version; otherwise REFUSE.- 26 of 28 have rc 0, and every one prints the crate version
Cop requirement for the 0.70.0 all-binaries gate (2026-09-24): ptop and score are built WITH their features (
-p aprender-present-terminal --features ptop,score), both on the nightly and at the release commit. A bin skipped for missing features counts as RED, not exempt. The manifest records each tool's feature set. This lands in the follow-up PR after #4206.Follow-up shape agreed with aprender-75 (#4218):
tools["apr@cuda"] = {asset, sha256, bin_sha256, version_sha, version_output, features:["cuda"], cuda_arch:"sm_121"}on aarch64-unknown-linux-gnu (gx10). Every tool gains afeatureslist (ptop/score: ["ptop","score"]). Until this lands, #4218 builds apr@cuda from the tag SHA and marks itmanifest_identity: absent.- added 6 commits that reference this issue
on Sep 24, 2026 #4189 follow-through: every [[bin]] on batch/0.70.0 @ c91ae0d (aprender-88, lambda-vector x86_64, CPU)
29/29 bin targets (28 names;
aprdeclared by apr-cli and the root facade) build (cargo build --release --locked -p <pkg> --bin <bin> [--features <required>], one bin at a time) and smoke (--versionexit 0 within 30s). Every--versionline carriesc91ae0dae. Negative control: a wrong SHA gets 0 matches.bin package build smoke sha in --version sha256[:16] alimentar aprender-data PASS PASS(version) yes 64ffdcea7d40c5d2apr apr-cli PASS PASS(version) yes 0d6099aacd9d9c2aapr aprender PASS PASS(version) yes 0dcc4f9dc30b55c8apr-corpus-ingest apr-cli PASS PASS(version) yes 49b341a064c45131apr-qa aprender-qa-cli PASS PASS(version) yes 65379470c4e539e0apr-qa-readme-sync aprender-qa-certify PASS PASS(version) yes bd7194328cdb4386aprender-cbtop aprender-cbtop PASS PASS(version) yes 36dccfbd44cc4c4eaprender-cgp aprender-cgp PASS PASS(version) yes 23fa435bff75e825aprender-compute-xtask aprender-compute-xtask PASS PASS(version) yes 145b6465ef629f36aprender-db aprender-db PASS PASS(version) yes 9f21620dbc62cd71aprender-explain aprender-explain PASS PASS(version) yes b90fb76921d51435aprender-orchestrate aprender-orchestrate PASS PASS(version) yes 3739f73e476654e0aprender-profile aprender-profile PASS PASS(version) yes 77ca7c2ee454933eaprender-ptx-debug aprender-ptx-debug PASS PASS(version) yes c55e7aa3c218267baprender-test-cli aprender-test-cli PASS PASS(version) yes 709cfa4faccd84f1aprender-train-bench aprender-train-bench PASS PASS(version) yes 4fdc63453ec5175faprender-train-distill aprender-train-distill PASS PASS(version) yes a140e728335950a0aprender-train-inspect aprender-train-inspect PASS PASS(version) yes 35a4ce881b5074fbaprender-train-lora aprender-train-lora PASS PASS(version) yes b1aa1122784edfeaaprender-train-shell aprender-train-shell PASS PASS(version) yes 175d2c8458869f74aprender-zram-generator aprender-zram-generator PASS PASS(version) yes 6fbf24fb3edf9d18presentar aprender-present-cli PASS PASS(version) yes bda51b6d76962e5dptop aprender-present-terminal PASS PASS(version) yes c107c5838e6bc01epv aprender-contracts-cli PASS PASS(version) yes 65c0a2860c235fcbscore aprender-present-terminal PASS PASS(version) yes 36b5c65733075e70simular aprender-simulate PASS PASS(version) yes 2c84d5eba8520ef8trueno-rag aprender-rag-cli PASS PASS(version) yes bd8cbc8d8504064etrueno-zram aprender-zram-cli PASS PASS(version) yes e71a8e3b1e6d12b7verificar aprender-verify-ml PASS PASS(version) yes cc26165b603cade9Scope: CPU x86_64 only. The CUDA
apr, aarch64 (gx10) and musl builds are not covered here. TSV: /mnt/nvme-raid0/cop-inbox/bins-4189-c91ae0dae.tsv#4189 verdict — batch/0.70.0 @ e260fe5: 29/29 [[bin]] targets PASS
Each bin:
cargo build --release --locked -p <pkg> --bin <bin>→ smoke--version(falls back to--help) → the short SHA must appear in the version line. build 29/29, smoke 29/29, sha_in_version 29/29.bin package build smoke sha alimentar aprender-data PASS PASS(version) yes apr apr-cli PASS PASS(version) yes apr aprender PASS PASS(version) yes apr-corpus-ingest apr-cli PASS PASS(version) yes apr-qa aprender-qa-cli PASS PASS(version) yes apr-qa-readme-sync aprender-qa-certify PASS PASS(version) yes aprender-cbtop aprender-cbtop PASS PASS(version) yes aprender-cgp aprender-cgp PASS PASS(version) yes aprender-compute-xtask aprender-compute-xtask PASS PASS(version) yes aprender-db aprender-db PASS PASS(version) yes aprender-explain aprender-explain PASS PASS(version) yes aprender-orchestrate aprender-orchestrate PASS PASS(version) yes aprender-profile aprender-profile PASS PASS(version) yes aprender-ptx-debug aprender-ptx-debug PASS PASS(version) yes aprender-test-cli aprender-test-cli PASS PASS(version) yes aprender-train-bench aprender-train-bench PASS PASS(version) yes aprender-train-distill aprender-train-distill PASS PASS(version) yes aprender-train-inspect aprender-train-inspect PASS PASS(version) yes aprender-train-lora aprender-train-lora PASS PASS(version) yes aprender-train-shell aprender-train-shell PASS PASS(version) yes aprender-zram-generator aprender-zram-generator PASS PASS(version) yes presentar aprender-present-cli PASS PASS(version) yes ptop aprender-present-terminal PASS PASS(version) yes pv aprender-contracts-cli PASS PASS(version) yes score aprender-present-terminal PASS PASS(version) yes simular aprender-simulate PASS PASS(version) yes trueno-rag aprender-rag-cli PASS PASS(version) yes trueno-zram aprender-zram-cli PASS PASS(version) yes verificar aprender-verify-ml PASS PASS(version) yes Nightly wiring: this check already exists in #4315 (batch/b2-ci-infra). Its nightly.yml builds every bin from
cargo metadata(nightly_manifest.py bins), and itsnightly_manifest.py recordturns a target red on a failed build, a failed--versionin an empty cwd, or a missing/wrong SHA (version-no-sha). The nightly is not wired until #4315 lands. It is blocked on a 6-file conflict with batch/0.70.0 (nightly.yml, ci.yml, binary-release.yml, check_model_ladder.sh, tree_reader_tests.txt, roadmap.yaml) and 5 red checks. Plan: take b2's nightly.yml whole; it supersedes my #4326 cuda-leg hunk.5 remaining items
- added 4 commits that reference this issue
on Sep 26, 2026 - added a commit that references this issue
on Sep 29, 2026 - removed sub-issues
on Oct 4, 2026
Operator, verbatim (relayed by the cop, aprender-cf, 2026-09-24): "fix these critical issues with aprender binaries" and "lambda-labs should always dogfood latest soveirgn stack nightly binaries".
This ticket is the PRODUCER side. #4186 is the consumer side: apr_bin.sh and pv_bin.sh refuse anything that isn't the latest green nightly. #4186 has nothing to consume until this lands, because only
apr(CPU) has a nightly today.Measured 2026-09-24 ~08:05Z (main @ aa7c6ef)
cargo metadata --no-depslists 28 distinct [[bin]] names (apris declared twice: by the root facade and by apr-cli).cargo build -p apr-cliNo space left on device, andrelease: needs: buildthen skipped the publish even though x86_64 was green. Last green: 09-23 @ 8c7822fnightly: apr-{x86_64,aarch64}-unknown-linux-gnu.tar.gz + .sha256, published 09-23 01:22Z, one day behind main--features cuda --bin aprcargo installonto the runnerNightly coverage: 1 of 28 bins, and it's red today. Every consumer except
aprbuilds its binary from source, or has no binary at all.Operator shape (verbatim via the cop, 2026-09-24): "nightly binary built is hard requirement for soveriegn stack, but intelligent: don't build if no work, only build working and passing CI and flag broken as tickets: this is arbiter job"
The arbiter (infra-8d) owns the decision; this repo's nightly.yml is the builder it drives. The gate, in order:
green_shain the publishednightly-manifest.json→ SKIP. No build, no ticket. This replaces nightly.yml's current "any commit in the last 24h" check, which rebuilds the same SHA and can't tell "no work" from "already built".ci / gate,workspace-test) are not allsuccesson that exact SHA → no build, no publish. One ticket, deduped by SHA: its title carries the short SHA, and the arbiter searches for it before opening another.--versionSHA doesn't match → no publish for that target. One ticket per SHA, listing every failing bin. The last green manifest stays in place, so consumers keep the last good build and never get a partial one.sha=<head>; nightly.yml refuses unlessgithub.sha == inputs.sha. This needs no change to the workflow's permissions. (b) nightly.yml checks for itself; reading check-runs needschecks: readadded to itspermissions:block, which is a permission change and so falls outside the standing yes. Recommended: (a).Fix (proposal). Workflow edit under the standing yes: quorum + green CI, no runs-on/secret/permission changes
rust:1.93.0-bullseyecontainer:cargo build --release --locked --workspace --bins(and--excludefor the GPU crates if they can't build CPU-only). Internal tools (e.g.aprender-compute-xtask,aprender-test-cli) go on an explicit denylist in the repo, not left out silently.cargo metadatabin set. A new[[bin]]added without a decision turns the nightly RED.nightly-manifest.jsonpublished as an asset of thenightlyrelease:{green_sha, built_at, run_id, tools: {<bin>: {target, sha256, version_output}}}. The job runs each binary's--versionand fails if the SHA it reports isn'tgithub.sha(proves the mechanism, not the label). This is the file P0: dogfood hosts run ONLY the latest green nightly apr/pv — apr_bin.sh + pv_bin.sh refuse anything older (arbiter manifest, fail closed) #4186's resolvers and the arbiter (infra-8d) consume on lambda.REDwith the run URL, so a consumer never mistakes a stale arch for current.aprgoes into the same manifest from the nightly x86_64 cuda build. It is RED rather than absent when that build fails.Out of scope, infra-side: the full gx10 disk that caused today's red (see memory: gx10 /mnt/nvme-raid0 is its root disk). Reported to infra-8d.
done_when
workflow_dispatchrun of the new nightly.yml is green on main and thenightlyrelease lists every non-denylisted bin for x86_64 (and aarch64 where built) +nightly-manifest.json.--versionSHA is wrong → RED.Related: #4186 (consumer), #3416 (nightly-red alerting), #4059 (dark CLI tests of kept binaries), #2582.