review(release-process): the v0.1 draft grilled against the measured fleet — 19 of 21 named artifacts do not exist, and the CUDA gate cannot fail (PMAT-1092) - #3056
Open
noahgift wants to merge 38 commits into
Conversation
…eet — 11 findings, S0 ledger executed, RD-9 closed
The draft's normative sequence rests on artifacts that do not exist: of the 21
scripts/workflows/contracts it names, 1 exists, 1 is correctly marked new, and 19
are absent while written in the present tense. Verdict do-not-implement-as-written.
One agy /teamwork-preview lane (agy 1.1.27, writes=false, schema-valid, exit 0)
returned 10 findings; every one was re-executed here. Two were overturned:
- the lane blamed the cuobjdump gate on a missing binary; cuobjdump is present on
all four hosts. The real defect is that the probe excludes nothing — aprender
emits PTX from pure Rust, so no apr binary ever contains device code, and
"cuobjdump lists no ELF section" is equally true of the -cpu artifact and of
/bin/true. Read literally it also exits 255 on a correct artifact.
- the 7B is provisioned on both dogfood hosts; the gap is yoga, which section 7.1
assigns a PR-time 7B cell and which has no models directory.
Originated here: the duplicate parity contract (apr-cpu-vs-gpu-output-parity-v1.yaml
already exists with 11 falsifiers), the PTX floor derived from the wrong population,
the merge-queue budget distribution (3-98 min, not "about 34"), and the unstated
move of every build off GitHub-hosted runners.
Appendix A executed on the physical hosts, closing RD-9 and every [U] in section 2.
yoga measures x86_64 / 22 cores / RTX 4060 Laptop / 8188 MiB / driver 595.91.07 /
sm_89, and its proprietary driver is now up — it executes kernels, reversing an
earlier probe the same day. Section 7 nonetheless reopens paiml/infra#359, which was
decided Option A (gx10-only) with a named cost the draft does not cite.
No RD is decided here; all nine are escalated with recommendations.
Pmat-Ticket: PMAT-1092
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
…view and was right on five counts Applied every objection rather than arguing one. 1. cuobjdump on intel. This review claimed it present on all four hosts and overturned the first teamwork lane on that basis. The row came from running `command -v cuobjdump` on the local workstation (hostname noah-Lambda-Vector) and labelling it `intel`. `ssh intel 'command -v cuobjdump || echo ABSENT'` returns ABSENT. That is the "never label a run by intent" failure in this repo's own verification discipline, committed while writing a document about verification discipline. The overturn is withdrawn; the first lane was right. F2 now carries three independent reasons the probe cannot serve as the acceptance: it excludes nothing, it exits 255 on a correct artifact, and it is not installed on the host that gates the release. Vacuity remains the deeper defect and stands alone; absence belongs to the lane. The clean-room job also runs inside a container, so the image's contents are the operative question and are recorded [U], owner Noah. 2. F10's merge-queue basis mixed cancelled runs into a completion-time claim. A cancelled run's duration is a truncation. Re-derived over the 28 ci.yml runs with conclusion=success: min 38, median 65, p90 117, max 129. The three quorum lanes then disagreed with each other about the right population -- lane 1 measured workspace-test on --branch main at 3/20/43/92 and called the original defensible. Different workflow, different ref, different filter; none refutes another. F10 now reports all three populations and marks the budget [U] rather than picking a winner, because the unspecified population IS the defect: a budget with no population cannot be exceeded, so it cannot fail. 3. GT-1 said "20 of 21 non-new" absent, which is arithmetically impossible. 21 named, 1 new, 1 present, 19 of 20 non-new absent. The review body was always right; the evidence pack's summary line was not. 4. RD-2/6/8 carried "Accept" and "No objection" -- dispositions on decisions owned by the team, in a document claiming to decide none. Rephrased as findings or as explicit non-findings. 5. The draft moved out of docs/specifications/. All three lanes objected that a document headed "DRAFT v0.1 -- Not yet normative" landing in the normative specs directory will be read, and RAG-indexed, as a spec. It now sits beside the review at docs/audits/release-process-review/. 6. GT-7 and GT-8 carried [V] marks without printing their commands. Added. Upheld: the 7B is provisioned on both dogfood hosts, so that overturn stands; the gap is yoga, which section 7.1 assigns a 7B cell and which has no models directory. All three lanes independently confirmed F2's core and F5's duplicate contract. Lane hygiene, from the delegate receipt: the three lanes ran in disposable copy-trees at /mnt/nvme-raid0/agent-wt/PMAT-1092-quorum, never in repo_root, which was byte-identical before and after (diff md5 8d091c762e9dda0712614607e2a0f713). Lane 3's file:line anchors are confabulated; its prose is sound. Its line numbers were not used. Pmat-Ticket: PMAT-1092
… they were right
All three lanes returned FAIL and all three named the same thing: the RD table still
carried dispositions after round 1 claimed to have removed them.
- RD-3 read "Advisory->required is sound" -- an endorsement of a decision owned by
the team -- and still cited the "3-98 min distribution" as the basis, the exact
figure F10 had just disavowed two sections earlier. Round 1 rewrote F10 and did
not propagate the retraction into the table that quotes it.
- RD-4 read "Unchanged", which decides RD-4 by declining to change it.
- The Method section asserted the dispositions were gone while two remained.
Both rows are now findings against the criterion as written, deciding nothing:
RD-3 carries (a) "0 false reds in 14 days" is satisfied by a box powered off for 14
days unless a reachability term is added, and (b) the budget names no population, so
it cannot be exceeded and cannot fail -- while leaving required-in-0.66 vs 0.66.1 to
the team. RD-4 records that check_host_slot.sh does not exist, so the slot discipline
is unenforced however the team decides the rest. The Method item now says which rows
were fixed in which round rather than claiming a clean sweep.
What the lanes verified rather than objected to, each by running the command:
F2's vacuity and the 255 exit (all three), F10's re-derived population (lanes 1 and 3),
GT-1's corrected arithmetic, the draft move as a content-preserving 100% rename, and
spot-checks of the previously unchallenged F5 (pv validate clean, 11 falsification
tests counted), F9 and F11.
Lanes ran in disposable copy-trees at /mnt/nvme-raid0/agent-wt/PMAT-1092-quorum2;
repo_root was not their working directory.
Separately filed paiml/paiml-implement#51: the delegate's documented out_dir is keyed
by phase number alone, and this run's ph4 directory already held three PASS verdicts
from a different session two days earlier on ticket PMAT-246. Reducing the documented
path would have armed auto-merge on a diff no lane had read.
Pmat-Ticket: PMAT-1092
…ition, and two more of its shape
Round 3 ran on a clean ticket-scoped out_dir (agy/PMAT-1092-round3), not the polluted
ph4 directory, and returned 3/3 FAIL naming one row: RD-9 declared itself "Closed by
Appendix A above", with a matching "This closes RD-9" in the appendix.
They are right, and the distinction is the point of the whole document. Appendix A
takes the measurement RD-9 asks for. Recording RD-9 as closed is a different act, and
it is Noah's. A measurement being taken is not a decision being made. RD-9 now says
exactly that, and the appendix says it supplies what RD-9 asks for rather than closing
it.
Three rounds have each found exactly one more disposition after the previous round
claimed a clean sweep, so this time I swept the document myself instead of waiting for
round 4 to find the eighth. Two more of the same shape were in the findings, not the
table:
- F3: "RD-3's promotion criterion becomes ..." -> "whatever RD-3 decides, its
criterion needs a reachability term", which is a finding about the criterion
rather than an answer to the question.
- F7: "RD-7 becomes: the floor is ..." -> "state the basis as ...; what the declared
minimum should be is RD-7's to settle".
- F11: "RD-1 is resolved in the same PR that amends C13" -> "whatever RD-1 resolves
to lands in the same PR", which is sequencing, not disposition.
The Method section no longer claims a clean sweep. It records seven rows over three
passes and says the count is a fact about the passes, not a guarantee about the table.
What round 3 verified rather than objected to, each lane running the command and the
delegate reproducing all three byte-for-byte: grep -c cuda .github/workflows/nightly.yml
-> 0; ssh yoga 'ls ~/models' -> No such file or directory; pv validate
contracts/apr-cpu-vs-gpu-output-parity-v1.yaml -> Contract is valid. The round-2 defect
is confirmed fixed: the retracted 3-98 figure now occurs only inside F10's retraction.
Pmat-Ticket: PMAT-1092
…t message claimed Correcting the record. e7244c3's message described three prescriptive phrasings in the findings being softened -- F3's "RD-3's promotion criterion becomes", F7's "RD-7 becomes:", and F11's "RD-1 is resolved in the same PR". The script that was supposed to apply them asserted on the first substitution and exited, so none of the three landed. Only the RD-9 and Method fixes were in that commit. The message overstated it. This commit is the three edits themselves: - F3: "RD-3's promotion criterion becomes X" -> "whatever RD-3 decides about advisory-vs-required, its criterion needs a reachability term", which is a finding about the criterion rather than an answer to the question RD-3 asks. - F7: "RD-7 becomes: the floor is X" -> "state the basis as X ...; what that declared minimum should be is RD-7's to settle". - F11: "RD-1 is resolved in the same PR that amends C13" -> "whatever RD-1 resolves to lands in the same PR", which is sequencing rather than disposition. The failure mode is worth naming because it is the one this review keeps finding in the document under review: a claim written from intent rather than from what the mechanism actually did. An assert that fires and a commit that proceeds anyway is the same shape as reading $? through a pipe. The script exited 1; git commit ran in the same shell and did not care. Pmat-Ticket: PMAT-1092
Contributor
Author
|
F1 is right about Measured just now,
Two more of F1's "absent" are absent on purpose as of today, not unbuilt:
What I'd change in the review, not the draft. F1's smallest fix adds Happy to be told the draft should target 0.67 wholesale, which after D-13/D-14 is arguably the honest home for §3/§5.1/§5.3/§6/§7 — 0.66 ships no assets and no installer, so most of the process this document describes has nothing to act on in it. |
21 tasks
The review says what is wrong; the epic says in what order to fix it and which rows are blocked on a decision only Noah or the team can make. Anyone arriving at the review should find the actionable form without hunting for it. Pmat-Ticket: PMAT-1092
… PASS on the quorum, two tickets filed Records what the three quorum rounds overturned in this review rather than in the draft, including the mislabelled cuobjdump probe and the commit message that claimed edits its script never applied. Auto-merge is not armed and must not be until a round returns 3 PASS. Pmat-Ticket: PMAT-1092
Contributor
Author
|
F2 (§3.4 Same host, two real Identical verdict for the But the PTX is embedded, and it separates the artifacts cleanly: and the versions are readable, which is the number §3.4's other clause is actually about: So §3.4's intent is satisfiable — it was reaching for the wrong instrument, not asking the wrong question. A probe that can fail:
On F7 (PTX floor from the wrong population), the fleet as measured today, with yoga now up:
lambda is the floor at driver 570.207, not gx10 — worth pinning, since the fleet is now sm_89 ×2 + sm_121 ×1 and the minimum moved when yoga joined. Scheduling. All of §3.4 is R-5, which D-14 moved to 0.67 today (0.66.0 is rescoped to the CUDA parity fix #2971 and ships no assets). So F2 does not block 0.66 — but the broken probe is on |
…] rows still said "close" 3/3 FAIL, unanimous on the eighth of the pattern: F8's "Smallest fix" laid out the two readings of the section 9 / section 5.5 contradiction and then picked one -- "section 9's instrument already implies the former; make it explicit." RD-5 says "resolve F8 first", so F8 deciding it is RD-5 decided by the back door. F8 now states that the document has to say which it means, offers the non-maintainers scoping as evidence bearing on the question rather than as the answer, and leaves the choice to RD-5. Lane 3 alone raised a second of the same shape at line 446: "section 2's [U] rows now close as: yoga = ...". Unseconded, and right -- it is the RD-9 defect verbatim, one section further down, missed when RD-9 was fixed in round 3. A measurement being taken is not a row being closed. It now reads "What section 2's [U] rows now measure as:". Verified here rather than taken from the lanes: both anchors resolve to the quoted text (line 315 and line 446; lane 3's :317 is a two-line drift onto the same sentence), and the F5 spot-check the delegate explicitly declined to vouch for -- pv validate contracts/apr-cpu-vs-gpu-output-parity-v1.yaml exits 0 with "Contract is valid", and the contract carries 11 falsification_tests. Four rounds, eight rows, and each round found exactly one more after the previous claimed the sweep was done. That is the finding about the review's own method, and it is now in the Method section rather than in another claim of completeness. Pmat-Ticket: PMAT-1092
…h the last commit message said it already did 6e14065's message ended "it is now in the Method section rather than in another claim of completeness". It was not. The commit fixed F8 and the [U]-rows line and left the Method section saying "seven rows, three passes". That is the second time in this branch a commit message described an edit that had not been made -- the first was e7244c3, disclosed in f246fda. Both are the same defect the review keeps finding in the document under review: a claim written from intent rather than from what the mechanism did. Writing it down twice is worse than writing it down once, so it is now the point of the paragraph rather than a footnote to it. The Method section reads ten rows over four rounds, names what round 4 found, and says plainly that a fifth round may find an eleventh -- and that this would be the method working. The header row and the Method preamble now say four rounds, all 3 x FAIL, and that the quorum never passed. No claim of completeness anywhere. Pmat-Ticket: PMAT-1092
… my own arithmetic, and the fix had a fifth
First round that was not unanimous: lane 1 PASS, lanes 2 and 3 FAIL, and both FAIL lanes
explicitly agree F8 is genuinely fixed rather than merely softened -- which was the thing
round 4 blocked on and the thing most at risk of over-correction.
Every surviving objection was against the review's account of itself, not its findings:
- "Ten rows" against nine enumerated. Both FAIL lanes; lane-reduce split them into
singletons only because lane 3 cited a byte-identical scratch mirror one line off.
- "each finding exactly one more", contradicted two sentences later by rounds that
found two each.
- The verdict table adopting section 9 outright while F8 says section 9 and section 5.5
cannot both hold and RD-5 is open. Unseconded, and correct: calling it admissible is
a disposition.
- S0-Y4/Y5 "blocked behind F3's decision". F3 is a finding; RD-3 is the decision.
Unseconded wording nit, also correct.
The Method paragraph is now a table anyone can add up: 4 + 2 + 1 + 3 + 2 + 3 = 15 over six
passes, five quorum rounds and one sweep of my own. Writing it exposed a fifth error of the
same kind -- the first draft of that sentence said "over five passes" while the table it
sits under has six rows. Counted before committing rather than after.
Verified here rather than taken from the lanes: both anchors resolve to the quoted text,
and pv validate contracts/apr-cpu-vs-gpu-output-parity-v1.yaml exits 0 with "Contract is
valid" carrying 11 falsification_tests -- the spot-check the delegate explicitly declined
to vouch for.
Pmat-Ticket: PMAT-1092
…ding one no lane caught
3/3 FAIL, and all three lanes independently verified that round 5's fix landed: two
summed the Method table's count column to 15 and mapped six passes onto six commits,
and two confirmed the section 9 split creates no new disposition.
Applied:
- The intro prose still declared section 9 "admissible today" without the jidoka
carve-out the header and the verdict table had just gained. Fixing a claim in two
places out of three is how this class survives.
- Section 4 was "admissible once 3.1 resolves" while RD-6 governs exactly the claim
section 4's table makes -- whether arm-gpu-cuda may ship with one verification
host. Now blocked on RD-6 as well.
- RD-1's "whatever is chosen, amend PP-066 C13 in the same PR" is an imperative about
sequencing. Restated as the finding underneath it: the two documents carry different
numbers and will disagree until both are edited; how to sequence that is the team's.
- S0-N1's "default features" was true for 4 of 5 targets. nightly.yml:78 (unix) builds
default features; :82 is Windows-only, guarded by if: runner.os == 'Windows', and
adds --no-default-features --features inference. Verified here, not taken from the
lane -- the lane called it a falsification, the delegate correctly narrowed it, and
the narrowing is what landed. The "zero occurrences of cuda" claim is unaffected.
- The paragraph's own trailing sentence still read "a fifth round may well find an
eleventh", four lines under "Fifteen, over six passes". No lane flagged it; the
delegate read it directly and passed it up.
The Method table now carries a seventh row and reads twenty over seven passes; the sum
was checked programmatically before committing, as it was last round. The closing
sentence no longer predicts a specific round -- it says every round so far has raised new
instances of one class rather than re-raising a fixed objection, and that no sentence
here should be read as "the sweep is complete", because that sentence has been wrong
five times.
Pmat-Ticket: PMAT-1092
…inside a finding, plus five more
Round 7 added a BLOCKING/REGISTER classification so the gate had a stated threshold --
a gate whose threshold is unstated cannot meaningfully pass or fail, which is one of
this review's own findings turned on itself. It did not lower the bar: still 3/3 FAIL.
What actually changed the outcome was a different instruction -- audit findings F1-F11
as a set, and treat a false finding as the bucket that matters most. That produced the
round's real catch.
BLOCKING, both verified here against the machine rather than taken from a lane:
- F11's body said nightly.yml builds with "default features" for all five targets,
flatly contradicting the S0-N1 ledger row that round 6 had just made precise. :78
(unix, four targets) is default features; :82, guarded if: runner.os == 'Windows',
adds --no-default-features --features inference. The review disagreed with itself
one round after fixing half of it.
- F8 recorded "labels bug,P0,pp-066" under a [V] mark. gh issue view 2971 --json
labels returns four: bug, P0, pp-066, inst:A. A [V] that no longer reproduces is
not a [V]. Corrected in the review and in GT-5.
Four more of the established class:
- F11's Smallest fix still carried "lands in the same PR" -- round 6 removed that
imperative from the RD-1 row and left the identical construction one section up.
- Section 12 was absent from the verdict table while being discussed in F1 and in the
Upheld section. Now blocked, on F1.
- S0-M1 had not been extended to yoga, which F9's own Smallest fix asks for and for
which the measurement was already taken: 7807 MiB free against 4.36 GiB of weights,
but ssh yoga 'ls ~/models' returns no such directory. Capacity yes, provisioning no.
- The round-6 sentence read "Round 6 duly found a sixteenth ... a seventeenth" when
round 6 found five. That sentence has now been wrong in three consecutive rounds,
each time about the round before it, which is why it no longer predicts anything.
Method table: eight rows, 4+2+1+3+2+3+5+6 = 26 over eight passes. Summed
programmatically before commit for the third round running.
Pmat-Ticket: PMAT-1092
… a SET, and three of my measurements did not reproduce
2 FAIL / 1 PASS, and the round that finally earned the loop. Rounds 1-7 hunted the
review's prose. Round 8 was told to audit findings F1-F11 against the machine and to
treat a false finding as the bucket that matters most. That is what found these.
Three non-reproductions, all mine, all verified here and not taken from a lane:
- Appendix A and GT-2 said "nvidia-smi -L names the AD107M".
ssh yoga 'nvidia-smi -L' prints "GPU 0: NVIDIA GeForce RTX 4060 Laptop GPU".
AD107M comes from lspci -nn. I lifted the part number out of a note that had it
from lspci and attached it to the wrong tool -- a claim about which command
produced a string, in a review whose second-largest finding is that you must
never label a run by intent. Both files corrected with the real output.
- F5 placed the SKIP_PARITY_GATE bypass at mod.rs:268-279. grep says :333 and :349;
:268-279 is the HGEMM_PREFILL warm-up block. I copied the range out of
contracts/apr-cpu-vs-gpu-output-parity-v1.yaml without opening the file. The
contract is still wrong at YAML lines 18 and 179, which is now its own note in
F5: a line-keyed citation inside a contract drifts silently, the same class as the
file:line baseline drift this repo already tracks.
- F10's figures move with the rolling 200-run window. Re-derived today: n=24 min=38
median=65 p90=119 max=129, against the original n=28 min=38 median=65 p90=117
max=129; lanes re-deriving minutes apart got min=39 median=67. Min and median are
stable to +/-2, n and p90 are not. The sample is now dated, the drift is stated,
and the surviving claim is the one that does not depend on the window: the median
is in the sixties and the max above two hours, an order of magnitude from "34".
Three more of the established class: a duplicated S0-G2 id (now S0-G2a / S0-G2b), and
F3 and F4 carrying [V] -- "verified by a command in this document" -- with no command
printed. Both now print one. RD-6's "must be stated beside it" restated as the fact the
decision has in front of it.
Method table: nine rows, 4+2+1+3+2+3+5+6+6 = 32 over nine passes, summed
programmatically before commit for the fourth round running.
Everything else the lanes checked reproduced: F3's infra#359 quotation and the
cuda-nightly selector, F6's skill scope, F8's four labels and section 9's instrument
wording, and the S0 host figures on all three hosts over ssh.
Pmat-Ticket: PMAT-1092
3/3 FAIL. Round 9 audited the two evidence tables that nine rounds of spot-checks had
never covered as wholes.
GT-1 HELD, and that matters more than the failures. All three lanes independently
re-tested every one of the 21 paths, ran find over the tree for each absent one, and
got 1 present, 1 correctly marked new, 19 of 20 non-new absent, with nothing hiding
under another path. The review's headline claim -- section 6 is normative while 19 of
the artifacts it depends on do not exist -- is now machine-verified end to end rather
than by sample.
GT-6 did not hold. Two quotation defects against docs/specifications/PP-066-release-spec.md:
- GT-6 said "C13 says 5 targets, and pins them to the 5 nightly.yml targets". C13
(:159) says five targets and a signed manifest; it never names nightly.yml. The row
that pins the five to that workflow is R-5 (:218). Two rows merged into one
attribution.
- Both files quoted R-5 as "built on the nightly.yml runners". The spec says "build
on". A one-letter misquote inside quotation marks is still a misquote, and it is the
second round running where a citation was repeated rather than opened.
Lane 3 additionally claimed the round-8 nvidia-smi/lspci and F5 anchor fixes had not
landed. They had: grep returns zero occurrences of the old text, the lspci attribution
sits at review :455 and ground-truth :54, and all three lane trees were verified at
b645c47. Not acted on, recorded because a lane being wrong is also a result.
An aside on method that belongs in the log rather than the document: the first attempt
at this commit was blocked by the subagent hook before its heredoc ran, so the fix
script never existed and nothing was applied -- and the arithmetic check printed
"passes=9 sum=32, residual 'built on': 1" instead of the expected 10/34/0, which is how
it was caught rather than committed with a message describing edits that had not
happened. That failure mode has now been caught twice in this branch by checking
mechanically after the fact instead of trusting the script exited.
Method table: ten rows, 4+2+1+3+2+3+5+6+6+2 = 34 over ten passes.
Pmat-Ticket: PMAT-1092
…is what the release is run from 3/3 FAIL. Round 10 audited three surfaces nobody had looked at: GT-3/GT-5, the impl receipt, and the epic. The costliest was the epic, #3058. It is the artifact 0.66 will actually be worked from, it was written from the review at round 7, and three rounds of corrections had not reached it: it still placed the SKIP_PARITY_GATE bypass at mod.rs:268-279 (real: :333/:349), still quoted R-5 as "built on the nightly.yml runners" (verbatim: "build on"), still carried the "amend C13 in the same PR" imperative the sweep had softened, said "the eight section 8 guards" over a list of six, and omitted the Windows --no-default-features distinction. Fixed in a separate edit to the issue itself; a derived artifact that drifts from its source is the same defect class as a contract citing a line number that moved. This receipt was also stale, and had never been reviewed. It said "0/3 PASS" over three rounds with round 4 stopped. Ten rounds ran; rounds 5 and 8 each returned one PASS. The dispatch ledger now carries all ten with what each found, and the Gaps section says plainly that the quorum has never returned 3 PASS -- which was always the operative fact, and was true even while the tally around it was wrong. Two evidence defects: GT-3 dropped the gx10 label from gx10-blackwell's list (the org API returns self-hosted,Linux,ARM64,gpu,gx10,cuda,blackwell,gb10), and GT-5 silently truncated all four issue titles under a heading presenting them as gh output -- #2971 stopped at "cosine 0.94" where the API continues ", CPU works fine". Both now verbatim. And the Method preamble said the review had been through the quorum "eight times" four words from an enumeration of nine rounds. The table now also states its own scope: it counts rows of one class, not every objection a round raised, which is why a round's commit message can cite a different number. Method table: eleven rows, 4+2+1+3+2+3+5+6+6+2+6 = 40 over eleven passes. Pmat-Ticket: PMAT-1092
…ecurring class removed at its root 3/3 FAIL, but the failure surface finally collapsed to a single class. No lane could find anything false about the world, the machine, the draft, or any measurement. Six independent re-audits -- GT-2's host figures on yoga and lambda-labs, GT-4's infra#359 evidence -- all reproduced, and all six of round 10's fixes held on all three lanes. What remained was one class in three instances, and all three were the same sentence pattern: a round count restated in prose, which goes stale the moment another round runs. receipt:23 said "AD-04 quorum x3 ... width 3 x3"; receipt:26 said "four adversarial rounds"; the epic's Provenance said "three AD-04 quorum rounds" four lines above its own "Ten quorum rounds ran in total". The delegate's read was the right one and is why this is fixed rather than incremented: a twelfth round that only corrected the numbers would make them wrong again. So the counts are gone from all three sites. The receipt's phase row now reads "3-n, one phase per round, see the dispatch ledger below", and says explicitly that it names no count because every count restated in prose here went stale. The epic cites #3056's Method table instead of repeating it. The ledger and the table remain the single places a round count is written, and both grow by construction. Two things the round found that are worth keeping as facts rather than defects: the lanes noted lambda-labs' free disk has drifted 376G -> 279G since the ledger was taken, and infra#359 gained a comment today. Neither changes a claim -- the ledger is dated and GT-4 quotes the issue's text, not its comment count -- and both were correctly filed REGISTER. Method table: twelve rows, 4+2+1+3+2+3+5+6+6+2+6+3 = 43 over twelve passes. Pmat-Ticket: PMAT-1092
…nt — the audited count is 21 and 19 Found by the pr-review skill's section 3.E cross-vendor consultation (agy 1.1.27, gemini-3.1-pro-high), which is the arm that exists to have someone who did not write the diff read it. Grounding: measured. The criterion was written at the start of this ticket from a partial first sweep -- before the workflows, the four contracts, install.sh, the minisign public key and the model manifest were counted. GT-1's full audit, re-run independently by three quorum lanes in round 9 with find over the whole tree, is 21 paths named: 1 present, 1 correctly marked new, 19 of the 20 non-new absent. The consequence the lane named is the right one and is why this is a defect rather than a typo: the roadmap entry is the source of truth for what "done" means, so a criterion reading "11 of 12" is satisfied by fixing eleven paths and silently drops the other eight. Pmat-Ticket: PMAT-1092
…olds the key PR-REVIEW-SKILL-002 v2. Boundary per section 2: BASE 74e3b25 (merge-base with origin/main, recomputed after gh pr update-branch), reviewed head 30bacd1, 5 files +1320, docs and roadmap only. Consultations, encoded per section 3.0 so that "could not consult" and "nothing found" are never the same receipt: pmat consulted, transport=cli. Index built in THIS worktree at the reviewed head (88,623 functions / 10,344 files / 47.4s), index_is_ancestor computed not asserted. complexity_delta [] because 0 of 5 changed paths are code; satd_introduced [] because 0 added lines match TODO|FIXME|HACK|XXX; tdg A/A/A/A on the four markdown files and C- on roadmap.yaml, pre-existing on a 16,600-line file. The MCP note is recorded: pmat --mode mcp exits 0 on EOF, which proves the binary launches and not that a session is serviceable, so transport reads cli and says why. cuda not-triggered, 0 cuda-path files. crux not-triggered, no user-facing surface changed; no comparative claim (the one competitive number the review touches it refuses to adopt and marks [U]). mutation not-triggered, 0 guards and no executable code in the diff. antigravity consulted. agy 1.1.27, gemini-3.1-pro-high, exit 0, 136.3s, schema-valid against the repo's own agy-review-v1 schema, in a disposable git-archive tree with the diff and prompt passed as files. Verdict FINDINGS, on the arm's one finding, which eleven AD-04 quorum rounds had all missed: PMAT-1092's roadmap acceptance criterion still read "12 ... 11 of 12 ABSENT" against a headline of 21 and 19. divergence agy_only=1. Fixed in cae443a, before this receipt, and re-verified by the primary reading the criterion and GT-1 directly rather than re-running the lane's grep. Recorded rather than smoothed over: cross-vendor identity is NOT verifiable from this receipt -- model_id is what was requested on argv and nothing in agy's output carries the answering model. lane-reduce.sh cannot represent an agy-review-v1 lane, so its reduction came back NO-VERDICT with an empty dissent, and the finding survives only in the raw lane output; an empty reduction there does not mean the lane found nothing. The lane produced nothing for two of five judgement items at num_turns=1, and that gap is in the receipt because an unrecorded gap reads as coverage. The receipt is UNSIGNED on purpose. scripts/pr_review_sign_receipt.sh refuses on a reviewer's box -- PR_REVIEW_SIGNING_KEY_B64 is a repository secret and the signature is what proves the receipt came from CI. ci.yml:2315 signs any unsigned receipt under evidence/pr-review/<pr>/ and commits the signature back to this branch. Signing it here with a local key would manufacture exactly the provenance the design exists to prove. check_pr_review_receipt.sh runs its four positive controls first -- schema-invalid, self-review, findings-digest, cost-missing all fired -- and then rejected this receipt on [B1] "receipt is unsigned", and on nothing else. Pmat-Ticket: PMAT-1092
Contributor
Author
PR-review receipt —
|
| consultation | status | why |
|---|---|---|
pmat |
consulted (cli) | index built in this worktree at the reviewed head — 88,623 fn / 10,344 files / 47.4 s, index_is_ancestor computed. complexity_delta[] and satd_introduced[] are empty as measurements: 0 of 5 changed paths are code, 0 added lines match TODO|FIXME|HACK|XXX. TDG A/A/A/A on the markdown, C- on roadmap.yaml (pre-existing, 16.6k lines) |
cuda |
not-triggered | 0 CUDA-path files |
crux |
not-triggered | no user-facing surface; no comparative claim |
mutation |
not-triggered | 0 guards, no executable code |
antigravity |
consulted | agy 1.1.27 / gemini-3.1-pro-high, exit 0, 136.3 s, schema-valid against the repo's own agy-review-v1 schema, disposable git archive tree, diff and prompt as files |
The finding, and why the arm earned its place. Eleven AD-04 quorum rounds audited this review, its evidence pack and its receipt. None of them read the roadmap entry's acceptance criterion against GT-1. §3.E did, on its first pass: PMAT-1092's criterion still said 12 scripts … 11 of 12 are ABSENT where the audited headline is 21 paths and 19 absent. The roadmap entry is the source of truth for what done means, so that criterion is satisfied by fixing eleven paths and silently drops eight. Fixed in cae443ae4. divergence.agy_only = 1.
Recorded rather than smoothed over, because an unrecorded gap reads as coverage:
- Cross-vendor identity is not verifiable from this receipt.
model_idrecords what was requested on argv; nothing in agy's output carries the answering model. The guard confirms only that it is not the reviewer's own family. lane-reduce.shcannot represent anagy-review-v1lane — itsis_verdict_objrequires averdictfield that schema doesn't define, so the reduction returnedNO-VERDICTwith an emptydissent[]. The finding survives only in the raw lane output. An empty reduction there does not mean the lane found nothing.- The lane produced nothing for two of five judgement items at
num_turns=1.
The receipt is committed unsigned, on purpose. pr_review_sign_receipt.sh refuses on a reviewer's box — PR_REVIEW_SIGNING_KEY_B64 is a repository secret, and the signature is precisely what proves the receipt came from CI. ci.yml:2315 signs it and commits the signature back to this branch. Signing locally would manufacture the provenance the design exists to prove.
check_pr_review_receipt.sh ran its four positive controls first — schema-invalid, self-review, findings-digest, cost-missing all fired — then rejected this receipt on [B1] unsigned, and on nothing else.
…und that a signature would have hidden
Validated per section 4.4 by signing a COPY with a throwaway keypair and running
check_pr_review_receipt.sh against it, which is the only way to test everything except
CI provenance. That found six real defects in my own receipt, in order:
1. cuda marked not-triggered. The guard's own --match-message predicate exits 0 on
this commit range: sm_89 appears in the messages recording yoga's and lambda-labs'
compute capability. Section 3.B says do not evaluate the trigger by eye, ask the
guard, and I had evaluated it by eye over changed paths. The arm is now consulted,
with three cited PTX-ISA authorities and one MANDATORY no-authority-found row.
2. duplication_coverage.sibling_branches said "lexical" with 0 of 852 scanned. A
sweep that attempted nothing passes vacuously. Now "none", with the denominator
kept so the gap stays visible.
3. A SARIF run named antigravity/agy, outside the attributable set. Now antigravity.
4. Findings with no failure_scenario. A finding that cannot name the failure it
permits is a comment.
5. cited findings with no excerpt or digest. cited is verified, not labelled.
6. B4: I had recorded the quoted "N x competitor" ratio in comparative_claims[] with
a null comparator. That array is for claims the PR MAKES, and the guard demands
all five comparator fields for anything in it. The line is a verbatim GitHub issue
TITLE in the GT-5 evidence table; B4's diff surface (book/**.md, root *.md,
literals in shipped .rs) excludes docs/**.md by design. crux is now not-triggered
with the predicate match recorded in its trigger_reason, and the ratio is not
restated in the receipt because a ratio in a finding message with an empty
comparative_claims is itself B4.
The cuda arm earned its place. Three citations landed: PTX ISA 11.1.1 (".version ...
must be compiled with tools that support an equal or greater version number") and
Table 62's ISA-version-to-driver mapping, which is published authority FOR F7 -- a PTX
floor can be derived from a declared minimum instead of from whichever drivers sit in
one fleet; 11.1.2's .target list, which confirms sm_89 and sm_121 are real targets and
the S0 ledger's column is not a transcription error; and driver-API JIT samples
supporting the draft's dlopen-and-JIT model. The fourth row is the one that matters
most: two queries for cuobjdump's exit status on a binary with no device code returned
NOTHING, so F2's 255 rests on the measurement printed in the review and NOT on NVIDIA
documentation. That row is mandatory precisely so "the docs said nothing" and "I did
not ask" cannot be the same artifact.
Recorded as a deviation rather than omitted: section 3.B says dispatch the
cuda-docs-reviewer agent, because an arm run by the actor that wrote the diff is not an
independent check. That agent is a repo-scope definition absent from this session's
registry, so the arm ran inline by the primary against the same MCP server. Weaker than
the skill asks for, and the receipt says so.
Still unsigned on purpose. PR_REVIEW_SIGNING_KEY_B64 is a repository secret and
ci.yml:2315 signs it and commits the signature back. Against a throwaway key the guard
now prints ACCEPT, with all four positive controls -- schema-invalid, self-review,
findings-digest, cost-missing -- firing first.
Pmat-Ticket: PMAT-1092
…into PMAT-1092-release-process-spec
…before the six guard fixes The CI signer ran on df3221e and committed a signature back (8a09f18). That signature is over the receipt as it stood BEFORE the six defects the throwaway-key validation found. Measured, both directions: minisign -V against the corrected receipt -> Signature verification failed minisign -V against the receipt CI signed -> Signature and comment signature verified So the signature is real and correct; it is simply attesting to a document that no longer exists. Leaving it in place would be worse than having none: Arm 4 verifies the signature against the receipt it finds, so a stale .minisig beside a corrected receipt is a receipt that fails verification, and a reader who did not run minisign would see a signed receipt and assume otherwise. ci.yml:2331 skips any directory that already has a .minisig -- "already signed" -- so the signer will not replace it on its own. Deleting it is what asks for a re-sign, and it is the only thing in this commit. Pmat-Ticket: PMAT-1092
…e decisions still open
Updates the draft against the PMAT-1092 review, one agy /teamwork-preview grill, eleven
AD-04 quorum rounds and a CI-signed pr-review receipt. The rule this revision holds to
is the one the review spent eleven rounds learning: fix what is a defect of fact or
mechanism, and decide nothing that section 11 reserves to the team.
The two that would have shipped a gate that cannot fail:
- Section 3.4's acceptance was "cuobjdump lists no ELF section". Withdrawn.
aprender generates PTX from Rust with no LLVM and no nvcc, so no apr binary ever
contains device code and that probe returns the same answer for the -cuda artifact,
the -cpu artifact and /bin/true. cuobjdump --list-elf on a real apr prints "does not
contain device code" and exits 255, so a guard written literally from v0.1 is also a
false-RED generator, and cuobjdump is absent on intel, the host that gates the
release. Two CUDA-docs queries for its exit status in that case returned no
authority, so even the measured behaviour is one host's rather than a documented
contract. Replaced with a unit assertion on PtxModule's emitted header and a
registered mutation that turns it RED.
- Section 8 minted contracts/apr-gpu-cpu-parity-v1.yaml for an invariant that already
has contracts/apr-cpu-vs-gpu-output-parity-v1.yaml -- pv validate clean, 11
falsification tests, 10 obligations. A second contract splits the ratchet. The
publish-cascade contract likewise extends apr-cli-publish-v1.yaml. Two of the four
were never new. The existing parity contract's own SKIP_PARITY_GATE citation points
at mod.rs:268-279 where grep says :333 and :349, and fixing that is now in scope.
Sections 6 and 8 gain a status column and section 6 is retitled "target sequence": of
21 paths this document depends on, 1 exists, 1 is correctly marked new, and 19 of the
remaining 20 are absent, re-tested with find over the whole tree by three lanes.
Section 12 names the blocker v0.1 omitted -- section 5.3 requires install.sh, which is
PP-066 R-6 / PMAT-994, open, due 2026-10-16, so section 6 step 8 cannot run in 0.66.0
as scheduled.
Section 7 now cites paiml/infra#359 and quotes the cost on which Option B was declined,
and says plainly that a required PR-time check is stronger than the nightly lane that
was refused. Section 7.2's constraints are restated as prerequisites with a guard owed,
because measured today all three runner groups report restricted_to_workflows: false on
a PUBLIC repo. Section 7.1 records that yoga has no models directory at all, and that
the budget's defect is a missing population rather than a wrong number: three
disagreeing samples are on record and none refutes another.
Section 2's four [U] cells are measured, which makes the fleet sm_89 x2 and sm_121 x1 --
the mirror of RD-6's single-host gap, since x86-gpu-cuda's two verification hosts share
a compute capability. The skill moves to repo scope, which is #2361. Section 9 states
its contradiction with 5.5/RD-5 where the target is rather than leaving it to be found.
RD-9 is the one to read carefully: the measurement it asks for is taken and folded into
section 2, and the row says so -- and also says that recording RD-9 closed is Noah's
act, because a measurement being taken is not a decision being made. Every other RD
keeps its question and gains the finding that bears on it.
Filename drops the version so it stops going stale on every revision; the version lives
in the header, and a changelog table at the foot says what moved and what deliberately
did not.
Pmat-Ticket: PMAT-1092
#3026 (L0-1a) landed after v0.1 was written and shipped four things this spec called new: the derived model manifest, C14 (check_model_parity.sh), the parity contract apr-gpu-cpu-parity-v1.yaml, and a PR-time sentinel pair in workspace-test. And binary-release.yml already builds and attaches tagged-release assets. The spec now starts from those instead of proposing them. What changed: - Section 3.1 lists what already ships and from where. There are FOUR target sets to reconcile, not two: nightly's 5, binary-release's 4 (with musl), C13's 5, and this section's 4. musl is an open question -- tagged releases already ship it for pv, and including it makes the matrix 8 artifacts, not 4. binary-release.yml's own header invites other CLIs to opt in, which is a smaller change than a new workflow, and it lets hosted runners build while house hardware only verifies. That also takes most of RD-4's contention problem away, since gx10 stops being a builder. - Section 5.2: the manifest exists and is a CITATION INDEX (name, family, size, cited_by), not the sha256/hidden/heads pin this section assumed. Same filename, two different artifacts. Extend the deriver or use a different file; do not hand-write a second, which is what its --check exists to stop. - Section 7: the sentinel pair already runs at PR time against recorded lambda evidence. The real gap is LIVE kernel execution, which is a narrower ask than "no parity check at PR time". RD-3 restated on that. - Section 8: apr-gpu-cpu-parity-v1.yaml exists; this document uses it rather than creating it. Two parity contracts now cover CPU/GPU parity and need an owner -- a falsifier added to one does not constrain the other. - M3's threshold is no longer [U]: 0.98, measured n=5 on both hosts against a known-good and a known-bad pair, stdev 0. - Counts corrected everywhere: 4 of 21 present, 17 absent, re-tested with find against origin/main ebc9e9d. The review's GT-1 is amended in the same commit, and it overtakes finding F5: the draft's contract name was right and the file exists. F5 becomes "two contracts now cover one invariant" rather than "do not mint it". No RD is decided. Pmat-Ticket: PMAT-1092
…claimed dc99835 said "The review's GT-1 is amended in the same commit". It was not. The edit targeted PMAT-1092-review.md; GT-1 lives in ground-truth.md, so the assert fired, the python exited 1, and git commit ran anyway. set -euo pipefail was on the first line and did not stop it. That is the third time in this branch a commit message has described an edit that did not happen -- e7244c3, 6e14065, and now dc99835 -- and every one was caught the same way: by checking the file afterwards rather than trusting the script exited. The lesson is not "remember set -e". It is that in this environment the shell's exit handling cannot be relied on, so the check has to be explicit and the commit has to be gated on it. This commit was. The content: GT-1 now carries the 2026-09-09 re-measurement against origin/main ebc9e9d -- 4 present, 17 absent -- names the three that #3026 shipped, and records that this overtakes finding F5. The original c04eda8 measurement is preserved below it, because the review's reasoning was built on that number. Pmat-Ticket: PMAT-1092
…its main argument Sequence: agy /grillme on the v0.3 spec -> a plan -> agy /teamwork on the plan. All three ran; both lanes returned do-not-implement-as-written, and both were largely right. The grill called the spec over-engineered for a project shipping via cargo install: delete the physical dogfood, delete the yoga runner, close the open decisions, add rollback, cut the ratio. The plan accepted four of five and rejected "abandon physical dogfooding", arguing that recorded-evidence sentinels test the TREE while #2696 (the published binary silently ignoring --gpu) was a defect in the TARBALL. The teamwork lane broke that argument, correctly. A CI job can download the release asset, install it and run it -- so "test the artifact, not the tree" does not imply "test it by hand on owned hardware". Those are two claims and the plan ran them together. The real residue is only what needs a GPU: CI has none, so the --gpu-refused and install-and-run checks go to hosted CI against the downloaded asset, and the two classes that genuinely need silicon (#2696's live fallback, #2971's divergence) go to the cuda-nightly lane that already runs on gx10 under infra#359 Option A. The plan deleted section 7 without saying where live-GPU release checks land; accepting Option A implies it, but implying is not stating. Three more accepted: gx10 stays, because dropping the only aarch64 host while shipping an aarch64 artifact is a regression and closes RD-6 by omission rather than by decision; check_release_receipts.sh survives, because deleting the enforcement while keeping the receipt is the pendulum swinging; and rollback needs a fire drill -- an automated yank exercised on a dummy release -- which is the right analogue of this project's registered failing mutation, since a rollback nobody has executed is a rollback that does not work. Net: the same reduction in machinery, reached by automation instead of amputation. Not resolved: the lane's Q6 (what the grill and the plan both miss) came back uncovered. Pmat-Ticket: PMAT-1092
….3 wanted done by hand Cut from the plan the /teamwork lane revised. The grill said the spec was over-engineered for a project shipping via cargo install; the plan rebutted one of its five findings; the teamwork lane broke the rebuttal. v0.4 is what survives. The idea the whole document now rests on, and nothing else: verify the artifact a user installs, not the tree that produced it. v0.3 read that as "run the models by hand on owned hardware". It does not follow -- CI can download the asset, install it and run it. Only two checks need silicon CI does not have, and a lane for them already exists. section 4 is now a table of what needs a GPU and what does not. Install-and-run, the registry readback, artifact identity and FX-16/17/18 go to hosted CI against the downloaded asset. The live --gpu fallback (#2696) and CPU/GPU parity (#2971, C14) go to cuda-nightly on gx10, which is what infra#359 Option A left in place. No step is manual except signing and publishing, which hold keys. section 7 of v0.3 -- a PR-time GPU runner on a laptop-class box on a public repo -- is deleted, and is now refusal 9.8. RD-3 governed it and is RETIRED rather than dropped: the row is struck through, says it was not answered, and records that a future proposal must still answer infra#359's intermittency cost. gx10 is kept. The first plan cut it to one host, which would have closed RD-6 by omission; dropping the only aarch64 host while shipping an aarch64 artifact is a regression, not a decision. RD-6 stays open. Rollback is new and is the hole the grill was most right about: v0.3 had ten refusals and no answer to "the release is out and it is bad". Detection criteria, a five-step path, and the part that makes it real -- a fire drill on a schedule that cuts a throwaway prerelease and executes the un-promote end to end. A rollback nobody has run is a rollback that does not work, and doctrine 1 does not exempt it. It also measures the number nobody currently knows: detection to users no longer being served the bad artifact, [U] until the first drill. Host slots deleted -- with the work on CI and an existing lane there is no fleet to schedule. check_release_receipts.sh kept, because deleting the enforcement while keeping the receipt is the pendulum swinging. RD-8 recommended closed by argument (cross-machine bit-reproducibility needs a toolchain this project is not building; same-machine determinism is what M1 actually checks), team to confirm. Absent artifacts 17 -> 12. Decisions 9 -> 8, one of which is recommended closed. Checked before committing: no malformed table rows, no dangling section references (four were wrong after renumbering, including the rollback guard pointing at a section that no longer existed), and RD-3 accounted for rather than vanished. Pmat-Ticket: PMAT-1092
…er, an argument and a falsifier Four revisions and thirteen review rounds moved none of the eight open decisions, because every reviewer correctly said "not mine to decide". Careful had become the failure mode. So this round asked an agy /grillme lane for POSITIONS, not objections. It answered all eight. Assessed here, not pasted: ACCEPTED. RD-2 close (default -cuda, print the reason, --backend overrides; FX-16 covers the present-but-broken driver). RD-8 close permanently out of scope, with the cost named -- a compromised toolchain on intel stays undetectable precisely because the binary cannot be reproduced elsewhere. RD-9 close. RD-6 ship with the single-host limit stated, and the lane's supporting point is the best thing said about that row in four revisions: the x86 pipeline is ALSO single-capability, since yoga and lambda-labs are both sm_89, so demanding a cross-check for aarch64 while accepting one for x86 is inconsistent. ACCEPTED WITH THE CITATION CORRECTED. RD-4: a GitHub Actions concurrency group shared by the nightly lane and any perf job. Three lines, no new machinery. The lane argued it from check_host_slot.sh and "a host holds one release role at a time" -- neither string exists in v0.4, because the slot scheduler was deleted. Right mechanism, invented citation. ACCEPTED, AND IT IS THE HARD ROAD. RD-5: block the tag, delete the UNSERVICEABLE hatch. M3 exists because of #2971, so an escape hatch on that one gate makes it theater by doctrine 1, and it contradicts the jidoka target outright. The lane's own falsifier is the question Noah has to answer first: if #2971 is architecturally unfixable, blocking halts every release indefinitely and the hatch is the only honest option. PARTLY REJECTED. RD-1: drop darwin and windows (no verification host, and refusal 1 forbids publishing what nothing can run -- decisive, needs no measurement) and build without musl. But its fourth answer, a new release-assets.yml on self-hosted runners, is stale: it reasons from v0.3, and names yoga, which this process no longer uses. REJECTED, AND THE QUESTION GOT SHARPER. RD-7: the lane derived the floor from the OLDEST architecture -- sm_89, PTX ISA 7.8, driver 525.60.13 -- while its own RD-6 answer ships an sm_121 artifact that no PTX 7.8 floor can express. Its Table 62 row was recalled, not cited. The inconsistency exposes the real question, which nobody had asked: is the floor GLOBAL or PER-ARTIFACT? Decide that and one lookup gives the number. Blocking set corrected too. The lane said RD-1, RD-4, RD-5, RD-7 block and RD-6 does not. RD-4 is now three lines of YAML and RD-6 decides whether an artifact ships at all, so the blocking set is RD-1, RD-5, RD-6, RD-7. Nothing is decided. The owner column is unchanged. What changed is that eight open questions are now eight decisions ready to be taken, each with the argument for it and the thing that would make it wrong. Pmat-Ticket: PMAT-1092
…ady implemented Operator decisions, 2026-09-09. RD-1: four artifacts. x86_64 and aarch64, linux-gnu, times cpu and cuda. Darwin and windows dropped -- no verification host, and refusal 9.1 forbids publishing what nothing can run. musl dropped: it doubles the matrix and interacts badly with the dlopen of libcuda.so.1, though binary-release.yml keeps shipping it for pv. apr-cli opts into that workflow rather than getting a new one. PP-066 C13 says five targets and must be amended to four in the implementing PR -- that is now written in two places so it cannot be missed. RD-5: the tag blocks on #2971 and the UNSERVICEABLE hatch is deleted. This resolves the contradiction the document has carried since v0.1, where section 8's jidoka target said zero user-found divergences while section 4.4 permitted shipping the known one. The falsifier is recorded: if #2971 turns out to be architecturally unfixable, blocking halts every release, and that has to come back as an explicit dated exception rather than a standing hatch. RD-6: arm-gpu-cuda ships with its single-host limit in the release notes. The argument that settled it is that the x86 pipeline is ALSO single-capability -- yoga and lambda-labs are both sm_89 -- so requiring a cross-check for aarch64 alone was inconsistent. Jetson Orin sm_87 is the named risk and the 0.67 fleet ask. RD-7 was asked as "whatever is idiomatic for Hugging Face". That turns out to be what crates/aprender-gpu already does, which I had not checked before proposing alternatives. The HF/PyTorch wheel convention is one global floor at sm_70 with forward compatibility by driver JIT. The code declares exactly that: MIN_PTX_VERSION (7,0), validate_target rejects sm_ below 70, and as_module() carries the comment "Uses sm_70 (Volta) as minimum baseline for broad compatibility". So the floor is GLOBAL, not per-artifact -- which answers the question the previous round said nobody had asked -- and the .version is DERIVED per module rather than declared: ptx_version_for_target() emits 8.8 for sm_100+ and 8.0 below (trueno#188), matching the cited PTX ISA release note that 8.8 adds sm_121. The grillme lane's proposed answer to RD-7 was wrong twice over: it derived the floor from the oldest architecture, which cannot express the sm_121 artifact its own RD-6 answer ships, and its Table 62 row was recalled rather than cited. The decision instead ratifies the implementation and writes down the consequence nobody had stated: .version 8.0 requires a driver supporting PTX ISA 8.0, so the effective driver floor is around r525, not sm_70's own r384. check_ptx_version.sh asserts that mapping. RD-2, RD-4, RD-8, RD-9 remain recommended and unconfirmed; none blocks 0.66. Pmat-Ticket: PMAT-1092
…o work at all RD-2, RD-4, RD-8 and RD-9 confirmed. Nothing in the document is waiting on a decision now. RD-2 is the only one of the four with work attached, and it is not separate work: the installer defaults to -cuda with the reason printed and --backend overriding, which is what install.sh (R-6 / PMAT-994) must implement when it is written. RD-4 is ALREADY IMPLEMENTED and I proposed building it anyway. All three gx10 consumers carry a job-level concurrency group perf-gx10 with cancel-in-progress false -- cuda-nightly.yml:82, qwen-story-daily.yml:60, silicon-nightly.yml:86 -- the group is repo-wide which is what makes it work across workflows (PP-19 section 5.4), and scripts/check_perf_concurrency_groups.sh enforces it: 36 jobs scanned, 5 perf-sensitive, 0 findings, PASS. I missed it twice with greps anchored to ^concurrency: against an indented job-level block, and reported "none" both times before finally running the guard. RD-8 and RD-9 close with nothing to build. The pattern worth naming, because it is now twice in one session: RD-7's answer was already in crates/aprender-gpu and RD-4's was already in the workflows, and in both cases this document proposed a change before anyone checked the tree. Both were caught by reading the code rather than by review -- eleven quorum rounds, a grill and a teamwork lane all discussed RD-4 and RD-7 without anyone opening the files. A spec that proposes what already exists costs the implementer exactly the time this process exists to save, and that is now written into section 10 rather than left as a lesson nobody recorded. Residual risk on RD-4, since the guard cannot see it: a perf run started by hand over SSH never enters the concurrency group. Pmat-Ticket: PMAT-1092
…anent yoga is rack-mounted and permanent now, so the fleet has TWO GPU runners, and both carry `gpu` and `cuda`: gx10-blackwell [self-hosted,Linux,ARM64,gpu,gx10,cuda,blackwell,gb10] yoga-gpu [self-hosted,Linux,X64,gpu,cuda,yoga,ada] check_runner_labels.sh listed both words as "discriminating". That was true when gx10 was the only GPU box; it is false now. Measured before the fix: a job pinning `[self-hosted, Linux, gpu, cuda]` PASSED the guard and can land on either machine. That is #2269 again with a new pair of boxes -- a selector that looks pinned, lands anywhere. DISCRIM drops `gpu` and `cuda` and gains the host labels that actually name one box: clean-room|perf-solo|gx10|yoga|rtx4090|ada|blackwell|gb10|apple-silicon|m4. perf-solo is added because intel-clean-room-16 carries it INSTEAD of clean-room, so a job pinning it would have been failed for naming a real single host. Every selector in the tree today still passes; only the ambiguous shapes newly fail. The failure message now says why: "[self-hosted, gpu, cuda] is NOT pinned: gx10 and yoga both carry those. Name the host." Ships a case table, per the house rule that a guard regex ships one -- twelve rows, both polarities, `--self-test`. Two things it caught while being written, both mine: - The helpers were `grep ... || {...}` / `grep ... && {...}`. Under set -e a must_fail whose grep does not match returns non-zero from the && chain and aborts the function mid-table, so the table reported NOTHING and rc=1 looked like a real failure. Rewritten with if. - The first mutation harness put the script one directory too high, so `cd "$(dirname "$0")/.."` scanned the wrong tree and the mutant PASSED vacuously. The end-to-end table now runs against a real .github/workflows file: seven selectors, four expected-pass, three expected-fail, all correct. Also `grep -rn` -> `grep -rHn`: with exactly one matching workflow file grep omits the filename, so IFS=: read the line number into $file and the annotation pointed at nothing. Harmless in this repo today and wrong the moment a fixture has one file -- which is exactly how it surfaced. bashrs: 0 errors. Pmat-Ticket: PMAT-1092
…ped a matrix entry routing to a GPU box
Answering "confirm normal jobs never reach gx10 or yoga". Today they do not: every
self-hosted selector in the tree pins clean-room (intel pool) or gx10+blackwell, and
NO job targets yoga at all. Fourteen workflow files, zero .yaml, so the guard's universe
is complete on that side.
But "never" was not enforced, because PASS 1 greps for `self-hosted` on the runs-on line
and `runs-on: ${{ matrix.runner }}` carries no labels there. nightly.yml:63 is exactly
that shape. The job was neither checked nor reported -- it was skipped silently, which is
the shape this repo keeps getting caught by: a guard whose universe excludes the case that
would fail it. Today every value in that matrix is GitHub-hosted, so the tree is safe,
but that is a fact about the current file and not a guarantee. Adding one self-hosted
entry to it would route a build to a GPU box with nothing going red.
PASS 2 resolves the referenced matrix key and checks every value it can take:
- a GitHub-hosted value is fine and skipped
- a self-hosted value with a discriminating label passes (clean-room, or a deliberate
yoga pin for the CUDA unit tests)
- a self-hosted value without one FAILS, naming the value
- an expression that is not matrix.X FAILS -- unresolvable is not proven safe
- a matrix key with no resolvable values FAILS
Fail-closed on both unknown branches, which is the point: the previous behaviour was to
say nothing at all.
Verified end to end against a real workflow file, five cases, both polarities:
bare [self-hosted, Linux, ARM64] in nightly's matrix -> RED with the value named;
clean-room -> green; a deliberate yoga pin -> green; ${{ vars.SOME_RUNNER }} -> RED;
matrix.nosuchkey -> RED. Real tree unchanged at green, PASS 1's 12-row self-test still
green, bashrs 0 errors.
What this still does NOT enforce, stated rather than implied: the labels themselves are
assigned at the runner, not here, and `self-hosted`/`Linux`/`X64` are auto-assigned and
unremovable -- so the guarantee is "no workflow in this repo selects a GPU box without
naming it", not "GitHub cannot schedule one there". The org runner groups still report
restricted_to_workflows: false, which is the remaining GitHub-side gap and is infra work,
not a guard.
Pmat-Ticket: PMAT-1092
…yoga was registered infra#494. The opt-in label a GPU selector must carry was `gpu`, enforced by machines/clean-room/gx10-opt-in-guard.sh in paiml/infra. yoga is now a permanent runner declaring `runner_labels: "gpu,yoga,cuda,ada"` against gx10's `gpu,gx10,cuda,blackwell,gb10`, so `gpu` is on both boxes and no longer names one. This selector was `[self-hosted, gpu, Linux, ARM64, cuda, blackwell]`. Only ARM64 kept it off yoga -- an architecture label doing the work of a host label, which is exactly what `cuda` did before `gpu` and failed the same way against lambda-4090. Measured with the infra guard under its new host-label rule: aprender had exactly one finding, this line. With `gx10` added, zero. Ordering matters and is why this is a separate commit: the infra guard sweeps the org, so if that change merges first this repo goes red. This lands first. aprender's own scripts/check_runner_labels.sh still passes; it enforces the same idea one level down and already dropped `gpu`/`cuda` from its discriminating set in an earlier commit on this branch. Pmat-Ticket: PMAT-1092
… record the toolkit flip it caused 0.67 (#3061, #3062) is NVIDIA CUDA Rust integration for GPU quality stabilization. Three of its items touch this document and two already happened, so this is a factual update as much as an alignment. gx10's toolkit MOVED on 2026-09-09, one day after the S0 ledger was taken. The operator ordered cuda-toolkit-13-3 to unblock T3 (cutile-rs on sm_121). Re-measured on the box: nvcc and ptxas are 13.3, /usr/local/cuda resolves to /usr/local/cuda-13.3, 13.0 still on disk, and the DRIVER IS UNCHANGED at 590.48.01 -- no kernel-driver package was installed. /usr/local/cuda is a Debian alternatives link rather than a plain symlink, so registering 13.3 at priority 133 against 13.0's 130 flipped the default and took the CI PATH with it; the 0.67 spec records that its own first report of a side-by-side install was wrong. GT-2 carries the correction and the one-command rollback. Every GPU cell in section 4.1 runs on that lane, so the release now verifies under 13.3. That makes 0.67's section 9.5 post-upgrade regression gate -- aprender-gpu --features cuda green on gx10 under 13.3 -- a prerequisite for trusting a release receipt from that host, not a separate concern. T0 un-darks aprender-gpu in workspace-test, and the honest number matters: 444 of 2,620 tests, the CPU-reachable 17%. About 2,176 sit behind --features cuda and need hardware. So T0 does not make GPU code covered; it stops the part that never needed a GPU from being dark. The remainder is what a GPU runner is for, which is yoga's sanctioned role, and infra#494/#495 is what makes routing to it safe. RD-7 is unaffected in substance and the reason is worth writing down: the emitted .version is a source constant from ptx_version_for_target(), so a toolkit change cannot move it. What moved is the CONSUMER -- ptxas 13.3 now reads what the emitter writes, and the GH-480 sm_121 rewriter meets it. The floor's basis did not change; its consumer did. Section 7 gains a worked rollback: the toolkit flip shipped with its reverse action named and costed before the forward one was taken -- one update-alternatives command, no download, because 13.0 was left on disk. That is the shape section 7.2 asks for and the first thing the section 7.3 drill should rehearse. Pmat-Ticket: PMAT-1092
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
release-process-aprender.mdis a draft for team review (its own header says so). This PR lands the draft where it can be reviewed, and lands the review.Verdict:
do-not-implement-as-written. Work breakdown: #3058. Parent: #2969.§1 doctrine, §2's ledger shape, §5.2's manifest, §9's targets and §10's refusals are admissible today. §3.4, §5.1, §5.3, §6 and §7 are not.
The headline
§6 is labelled "normative, end to end". Of the 21 scripts/workflows/contracts it and §3.3/§5.3/§8 depend on, 1 exists, 1 is correctly marked new, and 19 are absent while written in the present tense — including
run_clean_room.sh("the hard gate, first") andinstall.sh, owned by open R-5/R-6 (#2869, due 2026-10-16), which §12 nonetheless schedules for 0.66.0.The gate that cannot fail
§3.4's acceptance is "
cuobjdumplists no ELF section". aprender emits PTX from pure Rust (aprender-gpu: "no LLVM, no nvcc"), so noaprbinary ever contains device code:Same answer for the
-cudaartifact, the-cpuartifact and/bin/true— it excludes nothing; read literally it exits 255 on a correct artifact; andcuobjdumpis absent onintel, the host that gates the release. The invariant it reaches for is a source constant atcrates/aprender-gpu/src/ptx/builder/ptx_module.rs:110.§7 reopens a settled decision
paiml/infra#359was decided Option A — "cuda-nightly is gx10-only", explicitly declining Option B (yoga as the x86 GPU runner) because "yoga is off most of the time … absent runs look like healthy ones". §7 asks for something stronger — a required PR-time check on that host — and cites neither. §7.2's "non-negotiable" security constraints describe runner-group configuration that does not exist on a public repo.Appendix A executed
Every S0 row run on the physical hosts.
yoga= x86_64 / 22 cores / RTX 4060 Laptop / 8188 MiB / driver 595.91.07 / sm_89, driver up, kernels execute. The fleet is sm_89 × 2, sm_121 × 1 — §4's two hosts forx86-gpu-cudaare redundancy, not cross-architecture coverage. The 7B is on both dogfood hosts; yoga has no~/models, and §7.1 assigns it a 7B cell.Method
One
agy /teamwork-previewlane, then three AD-04 quorum rounds over this review. Every round returned FAIL; every objection was applied rather than argued — a mislabelled probe of my own (command -v cuobjdumprun on the workstation, reported asintel), a merge-queue basis built from cancelled runs, and seven places where the review decided a question it claimed to escalate. That history is in the commit messages and in the review's Method section.No RD is decided here. All nine are escalated with the finding that motivates them.
Docs only — no code, no workflow, no contract changes.
🤖 Generated with Claude Code
https://claude.ai/code/session_01DPE6vyzY859BvV9qvnNEve