fix(tests): apr-cli's integration surface goes 14 red -> 0 on clean main — three root causes, each fixed as a guard - #3053
Merged
Conversation
…the /100 wording #2394 removed `test_qa_016_validate_quality_score` FAILS ON CLEAN main (c04eda8) — reproduced in two independent worktrees. It demands `/100` or `points`; `apr validate --quality` prints TOTAL: 5/5 checks that ran (21 of 26 not implemented — not evidence of health) SCORE: 100% of the checks that ran Grade: A+ and that wording is deliberate. validate.rs::summary_line records why (#2394 finding 12): `✓ VALID 3/100 points` put a green badge next to what reads as 3%, against a denominator nothing was measured on — most of the checklist is Skip("Not implemented"). So `/100 points` is the RETIRED form and this test was the last thing in the tree still asking for it. It was invisible because `cli_integration` is not on ci.yml's `--test` line, so the target had never run in CI (the #2341 "integration targets never run" class). BSE-17's quick tier derives targets from the touched crates, so its first act was to surface a test that had been broken and unrun — working exactly as intended, and now blocking every apr-cli PR in the 0.66 chain. The rewrite is a GUARD for the fix rather than a relic of it: the SCORE line must state a denominator of checks that RAN (or `unavailable`), and `/100 points` must be ABSENT. Mutation: restore `SCORE: {pct}/100 points` in validate.rs -> RED ("Unexpected stdout, failed (var.contains(% of the checks that ran) || …)"); revert -> GREEN. Whole target: 83 passed, 0 failed, 2 ignored. Closes #3051 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH
noahgift
enabled auto-merge
September 8, 2026 12:43
|
§13.11 rung 1 — quorum shadow verdict Shadow mode: this records a verdict and merges nothing. A refusal |
…ain (#3051) Sweeping the class rather than the instance. `cargo nextest run -p apr-cli --tests` on CLEAN origin/main (c04eda8, second worktree) is 8037 passed / 14 FAILED. None of the 14 had ever run in CI: ci.yml names 7 of apr-cli's 64 integration targets on its --test line (#2341), and BSE-17's quick tier is the first thing to select them — so it turned the lights on, and every apr-cli PR in the 0.66 chain is red until these are fixed. Three root causes, each fixed as a GUARD for the thing that had drifted, never by loosening the assertion. 1. THE RETIRED SCORE WORDING (2 tests). cli_integration::test_qa_016_validate_quality_score and command_coverage::test_coverage_validate_quality_rich demanded `/100` / `points` — the wording #2394 finding 12 removed as dishonest (`✓ VALID 3/100 points` is a green badge next to what reads as 3%, against a denominator nothing was measured on). The second also offered `Score`, which never matched because the line reads `SCORE:`. Both now assert the honest contract — a score whose denominator is the checks that RAN — and require `/100 points` to be ABSENT. 2. STRINGIFIED OUTCOMES (10 tests, 30 assertions). Nine `*-lint --json` tests asserted `parsed[f].as_str().expect(..).contains("Ok")`. Measured, the surface has THREE shapes for one outcome field: {"status":"ok","efficiency":0.875} ddp-metrics-lint "Ok" prometheus-lint required_metrics "Ok { code: 134 }" nccl-diag-lint exit_code (a Rust Debug string in a JSON API) prometheus-lint emits two of the three in ONE response. `as_str()` returns None on the object shape, so the tests panicked in the `expect`. A shared `assert_outcome_ok` helper accepts all three and asserts the outcome IS ok — `.contains("Ok")` was a pass-grep in Rust clothing, since "NotOk" contains "Ok". The inconsistency itself is a finding about the --json surface and is filed separately rather than normalised away here. falsification_crux_i_04_rejects_stringified_arguments never reached its schema check at all: the command now requires --request-file in non-streaming mode, so it exited 5 on a USAGE error and the `!success` assertion passed for the wrong reason. It now passes the request and asserts the refusal is NOT the usage one. 3. A FIXTURE RACE (2 tests). pixel_regression's `test_apr_file()` creates a shared `playbooks/snapshots/test.apr` when missing, and nextest runs each test in its own PROCESS — five racing `fs::write`s to one path, and a reader seeing a partial file. Measured with the fixture deleted first, three runs: 4/1, 3/2, 4/1 failing a DIFFERENT test each time, which is why re-running one test never reproduced it. Publication is now write-temp-then-rename; four runs from a deleted fixture: 5/5, 5/5, 5/5, 5/5. `hex_dump.txt` was also genuinely stale — `apr hex` gained a header, decodes the gutter as the tensor's dtype (`0.0100`) instead of ASCII (`..#<`), and counts the remainder in ELEMENTS rather than bytes. Regenerated. The other three goldens were NOT committed: the comparator strips ANSI and their stripped content is byte-identical, so regenerating them was pure colour churn. After: `cargo nextest run -p apr-cli --tests` = 8051 run, 8051 passed, 21 skipped. Mutation (from the first commit, still standing): restore `SCORE: {pct}/100 points` in validate.rs -> RED; revert -> GREEN. Closes #3051 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH
This was referenced Sep 8, 2026
noahgift
added a commit
that referenced
this pull request
Sep 8, 2026
PMAT-1093 was cited by the kind-label commit and never existed — a Refs pointing at nothing, which is the small version of the defect pmat#1240 describes. Minted retroactively as `completed` with its acceptance command and `proof:` path. PMAT-1094 (#3055): the `*-lint --json` outcome surface emits three shapes for one field, one of them a Rust `Debug` string in a JSON API. Found sweeping #3051; the tests in #3053 accept all three deliberately and document the table, so this ticket's falsifier is the DELETION of `assert_outcome_ok`'s two string arms. Both by hand: `pmat work add` mints colliding ids (pmat#1169), and now pmat#1240 — two agents minting in parallel branches land on the same id and the merge deletes one, with `work validate` passing because uniqueness is preserved by the loss. roadmap 828 -> 830, sorted, unique; kind labels derived=103 missing=0 wrong=0. Refs #2873, #3055 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticket
Closes #3051. Not a PP-066 row — a main-red defect that blocks every apr-cli PR in the 0.66 chain (#3050, #3026, #3041).
Claim
apr validate --qualityprints a score whose denominator is the checks that ran, and the test says so — instead of demanding the/100 pointswording #2394 finding 12 deliberately removed.The red leg, on clean main
Reproduced in a second tree (
agent/F-1, same sha), so not a worktree artifact.The command prints:
crates/apr-cli/src/commands/validate.rs::summary_linerecords why that wording exists (#2394 finding 12):✓ VALID 3/100 pointsput a green badge next to what reads as 3%, against a denominator nothing was measured on. The producer is right and the test is stale — it was the last thing in the tree still asking for the retired form. Its comment was stale too ("scores 4/100 … exits non-zero"); the fixture scores 5/5 and the test asserts no exit code at all.Why it was invisible
workspace-testruns--libplus ONE explicit list of--testtargets inci.yml, andcli_integrationis not on that line — the target had never run in CI (#2341, "integration targets never run"). BSE-17 (#3044) added a quick tier that derives targets from the touched crates, so its first act was to surface a test that had been broken and unrun. That is the feature working, not a regression in it.The rewrite is a guard, not a relic
The assertion now requires the honest contract and forbids the retired one, so a regression to
/100 pointsturns this test RED:SCORE:line, reading…% of the checks that ranorSCORE: unavailable;checks that ran(ornothing was measuredwhen none did);/100 pointsabsent.Mutation (RED → GREEN)
SCORE: {pct:.0}/100 pointsinvalidate.rs, run the testFAILED. 0 passed; 1 failed—Unexpected stdout, failed (var.contains(% of the checks that ran) || var.contains(SCORE: unavailable))ok. 1 passed; 0 failedWhole target after the fix: 83 passed, 0 failed, 2 ignored — so this was the only breakage hiding in it. (One of the two ignored is
test_perf_inspect_latency, a wall-clock assertion; left alone, and it may never reach a required check per the repo's own rule.)Writes
crates/apr-cli/tests/cli_integration.rsonly.🤖 Generated with Claude Code
https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH