diff --git a/openspec/changes/import-crb-corpus/.openspec.yaml b/openspec/changes/archive/2026-08-08-import-crb-corpus/.openspec.yaml similarity index 100% rename from openspec/changes/import-crb-corpus/.openspec.yaml rename to openspec/changes/archive/2026-08-08-import-crb-corpus/.openspec.yaml diff --git a/openspec/changes/import-crb-corpus/design.md b/openspec/changes/archive/2026-08-08-import-crb-corpus/design.md similarity index 100% rename from openspec/changes/import-crb-corpus/design.md rename to openspec/changes/archive/2026-08-08-import-crb-corpus/design.md diff --git a/openspec/changes/import-crb-corpus/proposal.md b/openspec/changes/archive/2026-08-08-import-crb-corpus/proposal.md similarity index 100% rename from openspec/changes/import-crb-corpus/proposal.md rename to openspec/changes/archive/2026-08-08-import-crb-corpus/proposal.md diff --git a/openspec/changes/import-crb-corpus/specs/gold-eval/spec.md b/openspec/changes/archive/2026-08-08-import-crb-corpus/specs/gold-eval/spec.md similarity index 100% rename from openspec/changes/import-crb-corpus/specs/gold-eval/spec.md rename to openspec/changes/archive/2026-08-08-import-crb-corpus/specs/gold-eval/spec.md diff --git a/openspec/changes/import-crb-corpus/specs/parity-eval/spec.md b/openspec/changes/archive/2026-08-08-import-crb-corpus/specs/parity-eval/spec.md similarity index 100% rename from openspec/changes/import-crb-corpus/specs/parity-eval/spec.md rename to openspec/changes/archive/2026-08-08-import-crb-corpus/specs/parity-eval/spec.md diff --git a/openspec/changes/import-crb-corpus/tasks.md b/openspec/changes/archive/2026-08-08-import-crb-corpus/tasks.md similarity index 100% rename from openspec/changes/import-crb-corpus/tasks.md rename to openspec/changes/archive/2026-08-08-import-crb-corpus/tasks.md diff --git a/openspec/specs/gold-eval/spec.md b/openspec/specs/gold-eval/spec.md new file mode 100644 index 0000000..f144df8 --- /dev/null +++ b/openspec/specs/gold-eval/spec.md @@ -0,0 +1,79 @@ +# gold-eval + +## Purpose + +Candidate-only evaluation against gold-labeled corpus cases — absolute recall/precision and clean-case discipline, cheap enough to run on every meaningful pipeline change (no upstream review invocations). + +## Requirements + +### Requirement: Gold mode evaluates the candidate against labels without baseline runs + +The harness SHALL provide a gold-mode runner that, for a gold-labeled +corpus case, runs `/reviso:review` against the case's pinned checkout +(identically to parity mode's candidate leg) and judges the resulting +findings against the case's labels file — invoking no upstream review at +any point. Matching SHALL use the same matcher and correctness/cleanup +tiering as parity mode. + +#### Scenario: No upstream invocations + +- **WHEN** a gold-mode run completes +- **THEN** no `/code-review` (or other upstream review) invocation was + made; the only model costs are the candidate run and the judge's + matching calls + +#### Scenario: Same matcher, same calibration + +- **WHEN** gold mode matches candidate findings to gold labels +- **THEN** it uses the parity harness's matcher, so one calibration + covers both modes + +### Requirement: Gold metrics separate recall, proxy precision, and clean-case discipline + +Gold-mode output SHALL report per-case and aggregate: +`gold_recall_correctness` (matched correctness-tier gold issues ÷ total +correctness-tier gold issues), a precision proxy (candidate findings +matching any gold issue ÷ all candidate findings) explicitly labeled as a +proxy, and — for cases marked expected-clean — the count of candidate +findings, each of which is a false positive. Cleanup-tier gold issues the +candidate does not match SHALL be reported informationally and SHALL NOT +reduce gold recall. + +#### Scenario: Expected-clean case with findings + +- **WHEN** the candidate reports any finding on an expected-clean case +- **THEN** each such finding is counted and listed as a false positive + for that case, with no judge call required + +#### Scenario: Unmatched candidate finding is not auto-penalized + +- **WHEN** a candidate finding matches no gold issue +- **THEN** it lowers only the proxy-precision number and is listed for + potential promotion into the labels (real-but-unlabeled), mirroring + parity mode's claimed-wins handling + +#### Scenario: Cleanup-tier gold issue missed + +- **WHEN** a gold issue tiered cleanup has no candidate match +- **THEN** it appears in an informational bucket and does not change + `gold_recall_correctness` + +### Requirement: Synthetic cases run against materialized throwaway repos + +For corpus cases marked synthetic (no upstream repo), gold mode SHALL +materialize the case's diff into a freshly initialized throwaway git +repository (base content committed, diff applied as the change under +review) so the candidate reviews a real checkout; such cases SHALL be +skipped by parity tooling. + +#### Scenario: Synthetic case reviewed + +- **WHEN** gold mode runs a synthetic case +- **THEN** the candidate reviews a git checkout whose working diff equals + the fixture's diff, and the run is judged against the case's labels + like any other + +#### Scenario: Parity tooling skips synthetics + +- **WHEN** the parity baseline runner is pointed at a synthetic case +- **THEN** it refuses the case, identifying it as gold-mode-only diff --git a/openspec/specs/parity-eval/spec.md b/openspec/specs/parity-eval/spec.md index 53fb211..52a36e7 100644 --- a/openspec/specs/parity-eval/spec.md +++ b/openspec/specs/parity-eval/spec.md @@ -105,8 +105,14 @@ correctness tier, dismissal rate <10%. The corpus format SHALL support a public tier committed under `eval/corpus/` and a private tier referenced by local path and never -committed. Published eval runs (docs/evals.md) SHALL come from the public -tier only. +committed. Public-tier cases MAY carry a committed labels file under +`eval/corpus/labels/.json` in the shared label schema, with +imported labels attributed to their source (origin, upstream license +vendored beside the labels, import date); private-tier labels stay +outside the repo in the same schema. Published eval runs (docs/evals.md) +SHALL come from the public tier only, and SHALL distinguish gold-mode +metrics (whole corpus, per release) from parity metrics (active subset, +re-baseline cadence). #### Scenario: Private tier stays private @@ -114,6 +120,18 @@ tier only. - **THEN** no corpus entry, diff content, or finding text from it lands in the repository +#### Scenario: Imported labels carry provenance + +- **WHEN** a public case's labels were imported from an external benchmark +- **THEN** the labels file names its origin and the vendored upstream + license covers it + +#### Scenario: Published metrics are not conflated + +- **WHEN** docs/evals.md publishes results +- **THEN** gold-mode numbers and parity numbers are reported as distinct + metrics, each naming the corpus slice it covers + ### Requirement: Baseline runs record comparability metadata Each baseline run SHALL record, in a metadata artifact beside its outputs: @@ -181,3 +199,44 @@ against these labels before its parity numbers are trusted. - **WHEN** the calibration entry is added - **THEN** no diff content, finding text, or repository identity from it lands in this repository + +### Requirement: CRB import re-pins real PRs and skips unresolvable ones loudly + +The CRB importer SHALL derive each public corpus entry from the fixture's +source repo and PR number by resolving current base/head SHAs via the +GitHub API, and SHALL convert the fixture's gold issues into the shared +label schema. A fixture whose PR or SHAs cannot be resolved SHALL be +skipped with an explicit report line; the import SHALL never emit an +entry with unpinned or guessed SHAs, and SHALL NOT copy upstream diff +content into the repository. + +#### Scenario: Resolvable PR becomes a pinned entry + +- **WHEN** the importer processes a fixture whose PR still resolves +- **THEN** the emitted entry carries the resolved base/head SHAs and a + labels file derived from the fixture's gold issues + +#### Scenario: Unresolvable PR is skipped loudly + +- **WHEN** a fixture's PR no longer resolves to SHAs +- **THEN** the importer reports the skip by name and emits no entry for it + +### Requirement: The active parity subset is corpus data + +Corpus entries SHALL support an optional `active_parity` boolean; parity +baseline tooling SHALL run only entries marked true, while gold mode +ignores the marker. The imported corpus SHALL mark a subset (~10–15 +cases) spanning the corpus's languages and repos. (No CRB case is +expected-clean — the task-1.1 audit found all 50 carry gold issues — +so clean-case discipline is covered by gold mode's synthetic cases, not +the parity subset.) + +#### Scenario: Parity run filters on the marker + +- **WHEN** a parity sweep is launched over the public corpus +- **THEN** only `active_parity: true` entries incur baseline runs + +#### Scenario: Gold sweep covers everything + +- **WHEN** a gold-mode sweep is launched +- **THEN** every gold-labeled entry runs, regardless of `active_parity`