Skip to content

fix(tests): apr-cli's integration surface goes 14 red -> 0 on clean main — three root causes, each fixed as a guard - #3053

Merged
noahgift merged 2 commits into
mainfrom
fix/3051-qa016-honest-score
Sep 8, 2026
Merged

fix(tests): apr-cli's integration surface goes 14 red -> 0 on clean main — three root causes, each fixed as a guard#3053
noahgift merged 2 commits into
mainfrom
fix/3051-qa016-honest-score

Conversation

@noahgift

@noahgift noahgift commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Ticket

Closes #3051. Not a PP-066 row — a main-red defect that blocks every apr-cli PR in the 0.66 chain (#3050, #3026, #3041).

Claim

apr validate --quality prints a score whose denominator is the checks that ran, and the test says so — instead of demanding the /100 points wording #2394 finding 12 deliberately removed.

The red leg, on clean main

$ git worktree add --detach /mnt/nvme-raid0/agent-wt/premain origin/main   # c04eda87d
$ cargo test -p apr-cli --test cli_integration test_qa_016_validate_quality_score
test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 84 filtered out
Unexpected stdout, failed (var.contains(/100) || var.contains(points))

Reproduced in a second tree (agent/F-1, same sha), so not a worktree artifact.

The command prints:

  TOTAL: 5/5 checks that ran (21 of 26 not implemented — not evidence of health)
  SCORE: 100% of the checks that ran   Grade: A+

crates/apr-cli/src/commands/validate.rs::summary_line records why that wording exists (#2394 finding 12): ✓ VALID 3/100 points put a green badge next to what reads as 3%, against a denominator nothing was measured on. The producer is right and the test is stale — it was the last thing in the tree still asking for the retired form. Its comment was stale too ("scores 4/100 … exits non-zero"); the fixture scores 5/5 and the test asserts no exit code at all.

Why it was invisible

workspace-test runs --lib plus ONE explicit list of --test targets in ci.yml, and cli_integration is not on that line — the target had never run in CI (#2341, "integration targets never run"). BSE-17 (#3044) added a quick tier that derives targets from the touched crates, so its first act was to surface a test that had been broken and unrun. That is the feature working, not a regression in it.

The rewrite is a guard, not a relic

The assertion now requires the honest contract and forbids the retired one, so a regression to /100 points turns this test RED:

  • a SCORE: line, reading …% of the checks that ran or SCORE: unavailable;
  • a denominator of checks that ran (or nothing was measured when none did);
  • /100 points absent.

Mutation (RED → GREEN)

leg command result
RED restore SCORE: {pct:.0}/100 points in validate.rs, run the test FAILED. 0 passed; 1 failedUnexpected stdout, failed (var.contains(% of the checks that ran) || var.contains(SCORE: unavailable))
GREEN revert the mutant, run again ok. 1 passed; 0 failed

Whole target after the fix: 83 passed, 0 failed, 2 ignored — so this was the only breakage hiding in it. (One of the two ignored is test_perf_inspect_latency, a wall-clock assertion; left alone, and it may never reach a required check per the repo's own rule.)

Writes

crates/apr-cli/tests/cli_integration.rs only.

🤖 Generated with Claude Code

https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH

…the /100 wording #2394 removed

`test_qa_016_validate_quality_score` FAILS ON CLEAN main (c04eda8) — reproduced in two
independent worktrees. It demands `/100` or `points`; `apr validate --quality` prints

    TOTAL: 5/5 checks that ran (21 of 26 not implemented — not evidence of health)
    SCORE: 100% of the checks that ran   Grade: A+

and that wording is deliberate. validate.rs::summary_line records why (#2394 finding 12):
`✓ VALID 3/100 points` put a green badge next to what reads as 3%, against a denominator
nothing was measured on — most of the checklist is Skip("Not implemented"). So `/100
points` is the RETIRED form and this test was the last thing in the tree still asking
for it.

It was invisible because `cli_integration` is not on ci.yml's `--test` line, so the target
had never run in CI (the #2341 "integration targets never run" class). BSE-17's quick tier
derives targets from the touched crates, so its first act was to surface a test that had
been broken and unrun — working exactly as intended, and now blocking every apr-cli PR in
the 0.66 chain.

The rewrite is a GUARD for the fix rather than a relic of it: the SCORE line must state a
denominator of checks that RAN (or `unavailable`), and `/100 points` must be ABSENT.

Mutation: restore `SCORE: {pct}/100 points` in validate.rs -> RED
("Unexpected stdout, failed (var.contains(% of the checks that ran) || …)"); revert -> GREEN.
Whole target: 83 passed, 0 failed, 2 ignored.

Closes #3051

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH
@noahgift
noahgift enabled auto-merge September 8, 2026 12:43
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown

§13.11 rung 1 — quorum shadow verdict

S13-SHADOW pr=3053 head=0e4d8fc596a5794dca493c653c0fdb8af48e9b9a verdict=REFUSE class=Q1 arm_rc=1

Shadow mode: this records a verdict and merges nothing. A refusal
to arm is not a block (§13 adds zero rows to §7) — the pull request is
exactly as green as it was.

…ain (#3051)

Sweeping the class rather than the instance. `cargo nextest run -p apr-cli --tests` on
CLEAN origin/main (c04eda8, second worktree) is 8037 passed / 14 FAILED. None of the 14
had ever run in CI: ci.yml names 7 of apr-cli's 64 integration targets on its --test line
(#2341), and BSE-17's quick tier is the first thing to select them — so it turned the
lights on, and every apr-cli PR in the 0.66 chain is red until these are fixed.

Three root causes, each fixed as a GUARD for the thing that had drifted, never by
loosening the assertion.

1. THE RETIRED SCORE WORDING (2 tests). cli_integration::test_qa_016_validate_quality_score
   and command_coverage::test_coverage_validate_quality_rich demanded `/100` / `points` —
   the wording #2394 finding 12 removed as dishonest (`✓ VALID 3/100 points` is a green
   badge next to what reads as 3%, against a denominator nothing was measured on). The
   second also offered `Score`, which never matched because the line reads `SCORE:`. Both
   now assert the honest contract — a score whose denominator is the checks that RAN — and
   require `/100 points` to be ABSENT.

2. STRINGIFIED OUTCOMES (10 tests, 30 assertions). Nine `*-lint --json` tests asserted
   `parsed[f].as_str().expect(..).contains("Ok")`. Measured, the surface has THREE shapes
   for one outcome field:
       {"status":"ok","efficiency":0.875}   ddp-metrics-lint
       "Ok"                                 prometheus-lint required_metrics
       "Ok { code: 134 }"                   nccl-diag-lint exit_code   (a Rust Debug string
                                                                        in a JSON API)
   prometheus-lint emits two of the three in ONE response. `as_str()` returns None on the
   object shape, so the tests panicked in the `expect`. A shared `assert_outcome_ok` helper
   accepts all three and asserts the outcome IS ok — `.contains("Ok")` was a pass-grep in
   Rust clothing, since "NotOk" contains "Ok". The inconsistency itself is a finding about
   the --json surface and is filed separately rather than normalised away here.
   falsification_crux_i_04_rejects_stringified_arguments never reached its schema check at
   all: the command now requires --request-file in non-streaming mode, so it exited 5 on a
   USAGE error and the `!success` assertion passed for the wrong reason. It now passes the
   request and asserts the refusal is NOT the usage one.

3. A FIXTURE RACE (2 tests). pixel_regression's `test_apr_file()` creates a shared
   `playbooks/snapshots/test.apr` when missing, and nextest runs each test in its own
   PROCESS — five racing `fs::write`s to one path, and a reader seeing a partial file.
   Measured with the fixture deleted first, three runs: 4/1, 3/2, 4/1 failing a DIFFERENT
   test each time, which is why re-running one test never reproduced it. Publication is now
   write-temp-then-rename; four runs from a deleted fixture: 5/5, 5/5, 5/5, 5/5.
   `hex_dump.txt` was also genuinely stale — `apr hex` gained a header, decodes the gutter
   as the tensor's dtype (`0.0100`) instead of ASCII (`..#<`), and counts the remainder in
   ELEMENTS rather than bytes. Regenerated. The other three goldens were NOT committed: the
   comparator strips ANSI and their stripped content is byte-identical, so regenerating them
   was pure colour churn.

After: `cargo nextest run -p apr-cli --tests` = 8051 run, 8051 passed, 21 skipped.
Mutation (from the first commit, still standing): restore `SCORE: {pct}/100 points` in
validate.rs -> RED; revert -> GREEN.

Closes #3051

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH
@noahgift noahgift changed the title fix(tests): test_qa_016 asserts the honest score contract instead of the /100 wording #2394 removed fix(tests): apr-cli's integration surface goes 14 red -> 0 on clean main — three root causes, each fixed as a guard Sep 8, 2026
noahgift added a commit that referenced this pull request Sep 8, 2026
PMAT-1093 was cited by the kind-label commit and never existed — a Refs pointing at
nothing, which is the small version of the defect pmat#1240 describes. Minted retroactively
as `completed` with its acceptance command and `proof:` path.

PMAT-1094 (#3055): the `*-lint --json` outcome surface emits three shapes for one field,
one of them a Rust `Debug` string in a JSON API. Found sweeping #3051; the tests in #3053
accept all three deliberately and document the table, so this ticket's falsifier is the
DELETION of `assert_outcome_ok`'s two string arms.

Both by hand: `pmat work add` mints colliding ids (pmat#1169), and now pmat#1240 — two
agents minting in parallel branches land on the same id and the merge deletes one, with
`work validate` passing because uniqueness is preserved by the loss.

roadmap 828 -> 830, sorted, unique; kind labels derived=103 missing=0 wrong=0.

Refs #2873, #3055

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018RouwmUL7vFfJyCx9qLEoH
@noahgift
noahgift added this pull request to the merge queue Sep 8, 2026
Merged via the queue into main with commit d6ed9ac Sep 8, 2026
15 of 17 checks passed
@noahgift
noahgift deleted the fix/3051-qa016-honest-score branch September 8, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_qa_016_validate_quality_score fails on clean main: the assertion still demands the retired /100 points wording #2394 removed

1 participant