quorum receipt reports STALE for a correct receipt on some runners; the same head passes on rerun — the gate prints nothing to diagnose it

[noahgift]

What happened

The quorum receipt check sometimes reports STALE for a correct receipt at an unchanged head. Re-running the same job on a different runner passes, with no change to the head or the receipt.

PR	head	failed on	CI "tree" hash	passed on rerun
#563	5395900	intel-clean-room-3 (run 34988152076)	320a68de…	head moved on before a rerun
#569	56441f1	intel-clean-room-3 (run 35001751555)	1523c3a6…	intel-clean-room-4 (attempt 2)
#569	e91639e	intel-clean-room-11	57292577…	intel-clean-room-2 (attempt 2)

Locally, PRINT_HASH=1 scripts/quorum-gate.sh gives exactly the receipt's hash for each head. The CI hash matches no diff I could construct: I recomputed git diff <merge-base> <head> with the gate's own pathspecs against merge-bases cb94fc2, bfac33c, 8b2cced and ddd0c44, and none reproduces it. The same runners pass other heads (intel-clean-room-11 passed 09062f6). So it is not a fixed per-runner git config. It looks like per-run state in the persistent work directory.

Why it matters

A gate that reports STALE for a receipt the rerun accepts cannot be told apart from a real stale receipt, and the only way through is a rerun. That is flaky, which makes it red until the cause is found.

Done when

1. On STALE, the gate prints merge_base, BASE_REF's resolved sha, git rev-parse HEAD, and git diff --stat of what it hashed. The next occurrence then names its own cause instead of needing a rerun.
2. The cause is found and fixed. Suspects: a stale refs/remotes/origin/main in the runner's persistent clone (the fetch step's || git fetch origin main fallback updates FETCH_HEAD, not the remote-tracking ref); a leftover index or worktree from a previous job on that runner.
3. A test drives the gate against a clone whose origin/main lags the real base and asserts it refuses with the resolved shas named, not a bare STALE.

Found while landing PMAT-562/PMAT-564.

[noahgift]

Fourth occurrence, measured 2026-09-16 on PR #575 (the 1.31.0 cut).

head	e65a1fadf47173489a5180d8e9e0d8a1803cb735
receipt diff_sha256	206942314214add4142e97d6e289066d90fe2660
local PRINT_HASH=1 scripts/quorum-gate.sh	206942314214add4142e97d6e289066d90fe2660 — equal to the receipt
CI "tree" hash (run 35060619086, attempt 1)	229a1991ccc16bdb…
rerun, same head, same receipt, nothing changed	success

So the local gate and the receipt agree to the byte, and CI alone disagreed — then agreed on a rerun. That is the same shape as #563 and #569 (twice) and it is now four for four that a rerun clears it.

One new data point for whoever fixes this: this branch's diff is almost entirely NEW files under docs/audits/ and .quorum/evidence/, with no .rs at all, and it still reproduced. So whatever the runner-side state is, it is not specific to source files or to a particular merge-base.

Done-when item 1 of this issue — print merge_base, the resolved BASE_REF sha, git rev-parse HEAD and git diff --stat of what was hashed — would have made this five minutes of reading instead of a rerun and a guess.

[noahgift]

Fifth occurrence, 2026-09-16, PR #570 (PMAT-565), head a4e50143.

• receipt diff_sha256: 281ccc481b99a788…
• local PRINT_HASH=1 scripts/quorum-gate.sh: 281ccc481b99a788… — equal to the receipt, and the local gate passes
• CI "tree": 0b009ab379479af7…

That is five for five where the local gate and the receipt agree to the byte and CI alone disagrees. This one is on a branch that was force-pushed after a rebase onto the v1.31.0 cut, so the runner's persistent clone had the OLD tip of this same ref — which is the strongest hint yet that the cause is a stale remote-tracking ref or leftover index in the runner's work directory rather than anything about the diff.

[noahgift]

Sixth and seventh occurrences, both on PR #570 head bf4c9fcd — and this is the first time a rerun did NOT clear it, so the "just rerun" workaround is no longer reliable.

What I could rule out, by measurement rather than reasoning:

hypothesis	test	result
stale origin/main in the runner's persistent clone	recomputed the gate's hash with merge-base 583a58e, 8285616, 9884334, d324c57, cb94fc2	none produce CI's d785ba70…
different diff algorithm	histogram, patience, minimal	all produce the receipt's 9de10778…
different context or prefix	-U1, -U5, --no-prefix, --ignore-space-change	each produces a distinct hash, none is CI's
the trailing-newline trap	the gate captures with $(…) and hashes with printf '%s'; a naive pipe gives 33cd5552…	not the cause — the local gate and the receipt agree at 9de10778…

So CI is hashing a diff whose TEXT differs from anything this checkout can produce against any recent main, with the gate's own pathspec. That points at the content the runner has, not at how it diffs — which is what done-when clause 1 of this issue would settle in one line of output.

Measured context: CI checks out the PR merge commit (4c558797 = merge bf4c9fcd into 583a58ea), resets to bf4c9fcd, then runs git fetch origin main:refs/remotes/origin/main || git fetch origin main before the gate. origin/main is 583a58e and has not moved; the branch's merge-base is 583a58e; the receipt binds 583a58e.

Raising the cost estimate on this: it is now two reruns per PR and one PR that a rerun did not fix, on a REQUIRED check path. Clause 1 (print merge_base, the resolved BASE_REF sha, git rev-parse HEAD and git diff --stat of what was hashed) should ship before the next release rather than after.

[noahgift]

Eighth occurrence, PR #570 head 137cb17e — receipt fe4fd411…, local gate fe4fd411…, CI eec84777….

Additional hypotheses ruled out on this head, beyond the ones in the previous comment:

hypothesis	test	result
the runner's git ignores the :(exclude,glob) pathspec magic	recomputed with no excludes, with .pmat only, and with the whole .quorum/ tree excluded	ff820c88…, ff820c88…, fe4fd411… — none is CI's
the branch carries a mode change, a binary, or an odd filename	git diff --summary and --numstat over the whole diff	ten plain create mode 100644, no binaries, no non-ASCII paths

So on this branch CI is deterministic and wrong three attempts running, while every local reconstruction is deterministic and self-consistent. That combination is what makes clause 1 urgent: without the shas and the --stat of what CI hashed, there is no next experiment to run from outside the runner.

[noahgift]

Root cause measured on PR #583, head c54c73e, 2026-09-16.

The local gate and the CI gate hashed the same git diff and got different values: local b498dacb…, CI b9104ac4…. The diff text includes index <old>..<new> lines, and git abbreviates those object ids to an automatic length that scales with the clone's object count (core.abbrev unset). Re-hashing the same diff locally with the length pinned:

core.abbrev	hash
7	65355a5d…
8	b498dacb… (local, 39287 packed objects)
9	b9104ac4… (CI's fresh clone)
10	e901c95a…
40	164da1e5…

So diff_sha256 binds the clone that computed it, not the diff. A receipt that passes locally fails in CI, and the same receipt would fail locally once the local clone crosses the next abbreviation threshold.

Fix shape: git diff --no-abbrev (or -c core.abbrev=40) in scripts/quorum-gate.sh so the index lines are full ids on every clone, and a fixture that hashes one diff under two core.abbrev values and asserts equality — it goes red on the current script. The quorum-review skill's judged-diff.sh already strips ^index lines for the same reason. Not fixed in #583, which is docs-only on the triage rail; the receipt there carries the locally verified value.

[noahgift]

Falsifier and fix, named.

Fix: hash a diff whose object ids do not depend on the clone — git diff --full-index (or -c core.abbrev=40) in scripts/quorum-gate.sh before git hash-object --stdin, and the same in PRINT_HASH=1.

Falsifier (goes RED on today's script, GREEN after): one committed fixture diff hashed twice, once under -c core.abbrev=8 and once under -c core.abbrev=9, asserting equal output. The measured table on #583 head c54c73e is the expected pre-fix RED:

core.abbrev	hash
8 (local, 39287 objects)	b498dacb…
9 (CI fresh clone)	b9104ac4…
40	164da1e5…

Until this merges, the quorum receipt check is red on every PR whose author clone has a different object count than the runner, and it is not a required context, so nobody acts on it. Companion ticket on paiml/infra: make it deterministic or drop the check from the workflow, never leave it red-and-optional.

[noahgift]

Root cause found, and it is not a flake

Reproduced deterministically on PR #593 (head db49991e, base 537252d3), twice, with the CI job and the local gate disagreeing about a receipt whose content never changed.

scripts/quorum-gate.sh hashes git diff <merge-base> <head>. Every file in that output carries an index <a>..<b> <mode> line, and git abbreviates those blob hashes to whatever length keeps them unambiguous IN THE REPOSITORY IT IS RUNNING IN. A clone with more objects needs more characters. One commit range, byte-identical content on both sides, renders as different TEXT in two checkouts — and hashes differently.

Measured:

clone	objects	abbreviates to	hash
this workstation	42982	8	3617ae6b5c59180e...
clean-room runner	—	9	3d09e341e586e7b5...

git -c core.abbrev=9 locally reproduces the runner's hash exactly. A sweep gives six different hashes for one diff:

core.abbrev=7  -> 476acd3f2ae86af0...
core.abbrev=8  -> 3617ae6b5c59180e...   <- this workstation
core.abbrev=9  -> 3d09e341e586e7b5...   <- the runner
core.abbrev=10 -> 6e7551cd7e330031...
core.abbrev=12 -> 0e28893c1f179609...
core.abbrev=40 -> 31cfc5c23744f06c...

This explains both halves of the filed symptom

"the same head passes on rerun" — a rerun does not fix anything. It lands the job on a different runner, and it appears to pass whenever that runner's object count happens to agree with the author's. It is per-clone deterministic, which is exactly the shape that makes a bug look like infrastructure. On #593 it came up tails twice in a row.

"the gate prints nothing to diagnose it" — it does. quorum-gate.sh:484 prints receipt: <16 chars> and tree: <16 chars> on the two lines after the headline. Reading only the ✗ line is what makes it look silent. Those two numbers are what cracked this.

Fix

--full-index, which prints all 40 characters, so the text no longer depends on the repository. Verified identical under core.abbrev 7, 9 and 40 where there were three different hashes before.

Fixed in #593 with tests/falsification_quorum_hash_is_abbrev_independent.rs, which has two arms:

• it hashes one range under two abbrev settings and requires a single answer, and asserts the gate's diff_text= line carries the flag;
• an anti-vacuity arm requiring that without --full-index those same two settings DISAGREE — so a test that could never have observed the bug fails loudly rather than passing quietly.

RED proof: removing the flag gives test abbrev_independence_is_what_makes_the_binding_portable ... FAILED / test result: FAILED. 1 passed; 1 failed, with the anti-vacuity sibling still green.

Worth noting for #554

This is a different defect from #554 (the gate hashing a file the quorum itself writes). That one is already handled — .quorum/*.json and .pmat are excluded from the hashed diff. This one survived that exclusion because it is not about which files are hashed but about how git renders them.

[noahgift]

Two candidate remedies, and one of them has a collision

paiml-implement uses a different fix in skills/quorum-review/judged-diff.sh: pipe the diff through grep -v '^index ' before hashing. The argument for it is good — --full-index keeps the lines and merely makes them stable, so the hash stays coupled to object naming, whereas stripping decouples it entirely and makes the hash blind to index-only churn.

I tested it, and stripping is not safe for a binding hash.

For a binary file, git diff emits no content at all — the index line is the ONLY thing that distinguishes two different binary blobs:

diff --git a/blob.bin b/blob.bin
index 3f7a1c2..8b4e9d0 100644
Binary files a/blob.bin and b/blob.bin differ

Strip that line and every binary change renders identically. Measured in a scratch repo, two completely unrelated contents (BINARY-B vs BINARY-C-TOTALLY-DIFFERENT) against the same base:

remedy	change A	change B	distinguishes?
grep -v '^index '	620fa2e8c8e9e3bd…	620fa2e8c8e9e3bd…	NO — identical
--full-index	cbf094557fb7c3eb…	6207f0304362fbbc…	yes

So under the stripping remedy a receipt binds equally well to any binary content whatsoever. That is a collision by construction in the one property the binding exists to provide — it is the recycling this gate was written to prevent, reintroduced through a fix for something else.

Text-only diffs are unaffected, which is why it has held so far: judged-diff.sh has presumably never judged a branch that changed a binary fixture. That is a property of the branches seen, not of the fix.

Why --full-index is the one taken here

It preserves the discrimination and removes the variance, which is the whole requirement. The coupling to object naming that stripping avoids is not a defect on its own — a blob hash IS the content identity, and a binding hash should be coupled to content identity. What was wrong was never the coupling; it was that the RENDERING of that identity varied with the repository copy.

The generalisation worth keeping, from the infra session: this is a check whose correctness depended on a property of the copy rather than of the content being judged — the same class as paiml/infra#807. Any gate that hashes tool output should be asked the same question.

If the stripping approach is kept anywhere, it needs --binary alongside it so binary content reaches the hash as an actual patch rather than as a fixed English sentence. Otherwise the two remedies are not equivalent and only one of them binds.