Conversation
…ever substitutes the demo model, and a sharded SafeTensors index loads (#3022, #3024) apr chat silently loaded its built-in toy model for a sharded SafeTensors model: `Chat Demo (Tiny Model)`, `Loaded Demo format in 0.00s (0.0 MB)`, zero tokens, EXIT 0 — with the real model's path printed in the banner directly above. Nothing failed, so a user reasonably reads it as "my 7B produced an empty answer". ROOT CAUSE, and why one added arm is not the fix. `Path::extension()` returns the LAST dot-segment: on `model.safetensors.index.json` — the exact filename `apr pull` writes and then recommends — it is Some("json"), which matched no arm of detect_format and fell through to ModelFormat::Demo. That is the instance. The class is that TWO independent decisions existed: print_welcome_banner asked the PATH, ChatSession::new asked the first eight BYTES, nothing compared them, and so "banner says SafeTensors, session loads Demo" was reachable with no falsifier anywhere in the tree. * resolve_chat_format is the ONE decision: suffix before extension(), then the magic bytes of an odd-named file, then a REFUSAL. Demo is not an outcome for a path that exists — resolve_chat_model has already proven it does. * The refusal is CliError::ModelLoadFailed, exit 6, read from error.rs and never typed; it names the file, the recognised formats, and #3022. * Every consumer — banner, tokenizer, architecture detection, generator — is handed that one value, so the banner can no longer name a format the loader did not use. * ShardedSafeTensors reaches the transformer through the same three calls apr run already makes: load_from_index -> load_from_sibling -> convert_sharded. * The load line reports index.metadata.total_size. Printing the manifest's own ~20 KB would be the same lie under a different name: 0.0 MB. MEASURED, same fixture, two binaries built in this worktree: origin/main FAIL sharded x apr chat the DEMO model answered rc=1 this branch ok 4/4 cells, `Assistant: 2 + 2 equals 4.` at 3.2 tok/s rc=0 The three unaffected cells are GREEN on both, so the gate discriminates. CORRECTION to the report: `apr run` on a sharded index WORKS on main (measured, row 3). The defect is apr chat alone. Whatever failed for the reporter on a real 7B pull is not reproduced by this fixture and stays [U] — and the matrix now covers that cell on every run, so a regression there is caught rather than inferred. THE GATE (#3024). qwen-story-daily was green the same night, and structurally could not have caught this: `grep -c "apr chat" scripts/qwen-story.sh` is 0 and `grep -c index.json` is 0 — the command and the layout the defect needs are both absent. check_format_command_matrix.sh judges {sharded, single-file} x {run, chat} on four rules — the named format is never Demo, exit 0, tokens > 0, reported size > 0 — which is the GENERAL form of the defect, not the instance. make_sharded_safetensors.py derives the sharded fixture offline from a model the story already holds: a shard boundary is a property of the index and the split, not of the parameter count, so two shards of a 0.5B exercise weight_map, the per-shard header rewrite and the cross-shard lookup exactly as four shards of a 7B do — at 988 MB and seconds instead of a multi-gigabyte nightly fetch. Mutants (both RED, restored GREEN): the .safetensors.index.json arm never matches -> sharded_index_resolves_to_the_sharded_format_not_demo RED; resolve_chat_format returns Ok(Demo) -> three refusal rows RED. Contract contracts/patterns/format-command-honesty-v1.yaml (pv validate: 0 errors), 4 falsification tests, all wired: two case tables in ci / gate -> guard-tree, six unit rows in workspace-test, the live matrix on a host with a model. Refs #3022, #3024, PMAT-1080 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH
noahgift
added
pp-066
noahgift
enabled auto-merge
September 8, 2026 12:06
noahgift
added a commit
that referenced
this pull request
Sep 8, 2026
… (operator ruling 2026-09-08) F-1 (PMAT-1080, #3022, P0): `apr chat` silently loaded its built-in Demo model for a sharded SafeTensors index — exit 0, zero tokens, the real model's path printed in the banner. `Path::extension()` returns the last dot-segment, so on `model.safetensors.index.json` (the exact filename `apr pull` writes and recommends) it is Some("json") and matched no arm. Shipped in PR #3050 as a structural fix, not an added arm: `resolve_chat_format` is one decision — suffix before extension, then magic bytes, then a refusal from error.rs — and Demo is not an outcome for a path that exists. F-2 (PMAT-1081, #3024 ask 3, owner bse): the live matrix runs nightly against real models, and the two axes still uncovered — an `apr serve` column and a sharded-GGUF row (merge_gguf_shards, for which no fixture builder exists) — are named rather than dropped. The rows earn their place in a rescoped 18-row release by claim (1): apr reports truthfully and never silently substitutes what the user named. #3022 is the strongest instance of that failure in the tree — the substitution reported SUCCESS — and #3024 is why it survived: qwen-story-daily was green the same night and structurally could not have caught it (`grep -c "apr chat"` is 0, `grep -c index.json` is 0). DAG 103 -> 105 rows, 0.66 lane 32 -> 34; invariants PASS (violations=0); the spec's §5.0 block re-rendered byte-identical; roadmap 824 -> 826, sorted, unique, additive. Refs #3022, #3024, #2873 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
This was referenced Sep 8, 2026
…r, and sync the generated CONTRACT_COUNT
Two base-owned checks went red on the first push, both for the right reasons.
guard-tree / check_no_competing_harnesses.sh: PERF-009's predicate is textual — a script
that both DRIVES a model and COMPUTES A RATE is a second definition of how this project
measures itself. check_format_command_matrix.sh trips both halves purely through its own
must-match fixtures: `[0 tokens in 0.0s = 0.0 tok/s]` IS the defect row of its case table,
and its header names the axes `{apr run, apr chat}`. It derives no throughput and states no
comparator — it asserts that the format a command NAMES is not the demo model, exit 0,
tokens > 0 and reported size > 0. That is exactly the reason this detector already exempts
ITSELF and check_no_fabricated_baselines.sh, so it takes an allowlist entry with the reason
stated and a falsifiable condition for removing it ("if it ever derives a throughput or a
ratio, delete this line rather than widening the reason"), plus its own row in the
detector's case table so the exemption is covered: 18 passed, 0 broken.
Rewording the script to dodge the predicate was the alternative and was NOT taken: that is
the workaround PMAT-1074 already records as a known weakness of textual classifiers, and it
would leave the next reader with no statement of why the file is not a harness.
guard-cargo / FALSIFY-README-002: the generated CONTRACT_COUNT block stated 1815 against a
merge tree carrying 1816 — this row adds contracts/patterns/format-command-honesty-v1.yaml.
A generated number is an EQUALITY, not a ratchet, so `make readme-sync`.
Refs #3022, #3024, PMAT-1080
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH
…rries no machine path
Two base-owned guards, both right.
check_no_claim_literals.sh (SPEC-2.0's claims ratchet, shrink-only, 452 known + 1 new):
the receipt quoted the defect's own banner including `= 0.0 tok/s` and the fix's including
`= 3.2 tok/s`. The guard cannot tell a quoted defect from a boast, and it should not have
to: 0.66 delivers no speed, so a tok/s number on a doc surface is a claim this release does
not make. Neither number was load-bearing either — the defect is "zero tokens generated,
exit 0", and a rate is redundant once the token count is zero. The receipt now states the
token counts and drops the rates; every fact survives.
check_hardcoded_paths.sh: `.pr/F-1/accept.sh` hardcoded /home/noah/.cargo/bin/cargo, the
same finding L0-1a's accept.sh carried. `CARGO="${CARGO:-$HOME/.cargo/bin/cargo}"` keeps
the pin (never a bare `cargo`: a shell function of that name overrides CARGO_TARGET_DIR)
without the machine path. Local: claims PASS, hardcoded-paths delta +0.
Refs #3022, #3024, PMAT-1080
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH
…k takes the union Both sides add steps to the same job and neither replaces the other: main carries L0-1a's three manifest steps, this branch carries F-1's two case tables. Verified by counting each step after the resolution rather than by eye — one occurrence of each of the four. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticket
PMAT-1080 · PP-066 row F-1 · closes #3022, closes #3024 (asks 1, 2, 4; ask 3 is row F-2) · epic #2873
Claim
A model file the user named is the model that answers, or the command refuses.
apr chatno longer substitutes its built-in toy demo model for a sharded SafeTensors index, and amodel.safetensors.index.jsonnow loads the real model.The defect, measured on
origin/mainbefore anything changedBinary
apr 0.65.2 (c04eda87d), built in this worktree from the tip ofmain. Model: a 2-shard fixture derived offline from~/models/qwen2.5-coder-0.5b-instruct-safetensors— no download.origin/mainapr runapr chatLoaded SafeTensors format in 0.50s (988.1 MB)apr runOutput: 2 + 2 equals 4.apr chatChat Demo (Tiny Model)·Loaded Demo format in 0.00s (0.0 MB)·[0 tokens in 0.0s]· exit 0Model Chat (Sharded SafeTensors)·988.1 MB·[12 tokens in 3.7s = 3.2 tok/s]·Assistant: 2 + 2 equals 4.Records:
evidence/format-honesty/{before,after}/.One correction to #3022's framing.
apr runon a sharded index works onmain— row 3, measured, unmodified. The defect isapr chatalone. Whatever failed for the reporter on a real 7B pull is not reproduced by this fixture and stays[U]; the matrix now covers that cell on every run, so a regression there is caught rather than inferred.Root cause — and why one added arm would not be the fix
Path::extension()returns the LAST dot-segment: onmodel.safetensors.index.json(the exact filenameapr pullwrites and then recommends) it isSome("json"), matching no arm and falling through toModelFormat::Demo. That is the instance.The class is that two independent decisions existed —
print_welcome_bannerasked the PATH,ChatSession::newasked the first eight BYTES, nothing compared them — so "banner says SafeTensors, session loads Demo" was a reachable state with no falsifier anywhere in the tree.resolve_chat_formatis now the one decision: suffix beforeextension(), then the magic bytes of an odd-named file, then a refusal (CliError::ModelLoadFailed, exit 6, read fromerror.rs).Demois not an outcome for a path that exists.RED test sha
9d0ddcf31—crates/apr-cli/src/commands/chat_format_honesty_tests.rs, 6 rows, both polarities.Acceptance (
.pr/F-1/accept.sh, 6/6)Mutation (RED → GREEN), run locally, both mutants
detect_format's.safetensors.index.jsonarm never matchessharded_index_resolves_to_the_sharded_format_not_demo(3 passed / 1 failed)resolve_chat_formatreturnsOk(Demo)instead of refusinga_truncated_file_is_refused…,an_existing_unrecognised_file_is_refused_not_demoted,resolve_never_answers_demo_for_a_file_the_user_named(1 passed / 3 failed)And the gate's own RED leg — the whole point of #3024 — same matrix, same fixture, two binaries:
The three unaffected cells stay GREEN on the defective build, so the gate discriminates rather than merely failing. CI run ids follow on this PR's first green.
Contract
contracts/patterns/format-command-honesty-v1.yaml—kind: pattern, 4 equations, 5 proof obligations, 4 falsification tests.pv validate:0 error(s), 0 warning(s).The fixture (#3024's "deliberately kept small")
scripts/make_sharded_safetensors.pysplits onemodel.safetensorsinto N shards plus a HuggingFace index — offline, deterministic, no dependencies. A shard boundary is a property of the index and the split, not of the parameter count: two shards of a 0.5B exerciseweight_map, the per-shard header rewrite and the cross-shard lookup exactly as four shards of a 7B do, at 988 MB and seconds instead of a multi-gigabyte nightly fetch. Its 7-row case table proves every tensor's bytes survive, the index names exactly N shards, a second run is byte-identical, and--shards 1/ a non-safetensors input are refused — because a fixture that is not actually sharded would make a green matrix prove nothing.Quorum
Single-lane (a bounded CLI-correctness row with an executable falsifier on both polarities and a measured RED leg on
origin/main); no design fork was open.Receipt
docs/audits/impl-PMAT-1080-receipt.mdWrites
crates/apr-cli/src/commands/chat*.rs·crates/apr-cli/src/commands/chat_format_honesty_tests.rs(new) ·scripts/check_format_command_matrix.sh(new) ·scripts/make_sharded_safetensors.py(new) ·contracts/patterns/format-command-honesty-v1.yaml(new) ·.github/workflows/ci.yml(two guard-tree steps, both cargo-free) ·evidence/format-honesty/·docs/audits/impl-PMAT-1080-receipt.md·.pr/F-1/Not in this row
merge_gguf_shards) andapr servecells — named in F-2; no fixture builder exists for sharded GGUF and none is invented here.🤖 Generated with Claude Code
https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH