Re-aim the parity eval at built-in /code-review medium - #10
Merged
Conversation
Upstream /review is an effort-scaled skill embedded in the CLI binary, not the marketplace recipe Reviso forked; its default level (xhigh) is a recall-biased deep audit opposed to Reviso's precision invariant, and the old hash-watch tracked a dead file. Re-aim the harness: - Baseline = /code-review pinned to medium, fan-out allowed, majority-of-3; runs record identity (CLI version, level, resolved models) and refuse degraded or mislevelled runs. CLI version roll = re-baseline event. - Findings harvested from the typed ReportFindings call in the run transcript (report-findings.sh); prose extraction is the fallback. - Parity and P0 misses scoped to correctness-tier baseline findings; cleanup tier is informational. Tier calibration: 95% agreement on the first hand-labeled case; matcher spot-check 7/7 incl. a contradictory-root-cause trap. - eval/reference: behavioral drift detection + local-only skill extraction (extract-builtin.sh; fingerprint stable across 2.1.219-223); the 2026-08-03 marketplace snapshot is marked superseded, kept as history. - Prior runs annotated SUPERSEDED (identity unreconstructable); docs updated; eval/runs/private and eval/reference/private gitignored. Also carries the import-crb-corpus OpenSpec proposal (planning artifacts only, no implementation). Signed-off-by: Michael Hohlios <michael@vyttle.com>
Signed-off-by: Michael Hohlios <michael@vyttle.com>
MHohlios
added a commit
that referenced
this pull request
Aug 6, 2026
Follow-up to #10: the changelog entry was pushed while that PR was being merged and missed the squash. Adds the [Unreleased] section documenting the eval re-aim (repo-side only; plugin surface unchanged, no version bump). Signed-off-by: Michael Hohlios <michael@vyttle.com>
MHohlios
added a commit
that referenced
this pull request
Aug 6, 2026
Full cleanup of the OpenSpec state now that #6 and #10 have both landed: - `openspec/specs/` gains the five capability specs (`review-command`, `review-pipeline`, `deterministic-detectors`, `mock-pr-assembly`, `parity-eval`), composed from `add-reviso-review`'s ADDED requirements with `re-aim-parity-eval`'s delta applied to `parity-eval`. All five pass `openspec validate --specs`. - Both completed changes move to `openspec/changes/archive/2026-08-06-*`; `import-crb-corpus` remains the only active change. - markdownlint and internal links verified locally (the only on-disk-unresolvable links are the pre-existing GitHub-relative ones in `.lycheeignore`). Signed-off-by: Michael Hohlios <michael@vyttle.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Upstream
/reviewturned out to be an effort-scaled skill embedded in the CLI binary — not the marketplace recipe the parity harness was forked from — and its default level (xhigh) is a recall-biased deep audit philosophically opposed to Reviso's precision invariant. This PR re-aims the eval harness at the coherent target: built-in/code-reviewpinned to medium, scored on correctness-tier findings only.meta.json: CLI version, level, resolved models), rejects degraded/mislevelled runs. A CLI version roll is a re-baseline event;judge.shrefuses cross-version comparisons.ReportFindingscall harvested from the run transcript (report-findings.sh, verified against a live smoke run); prose extraction is the fallback.eval/calibration/README.md; labels stay in the private corpus).extract-builtin.shproduces a local-only skill dump + committed fingerprint (stable across CLI 2.1.219–223). The 2026-08-03 snapshot is marked superseded, kept as history.SUPERSEDED; docs updated; private dirs gitignored.Also carries the
import-crb-corpusOpenSpec proposal (planning artifacts only).Testing
/reviso:reviewon this branch — its two surviving bug findings are fixed in this commit.