openspec: sync gold-eval and parity-eval specs, archive import-crb-corpus

openspec/specs/gold-eval/spec.md

@@ -0,0 +1,79 @@
+# gold-eval
+
+## Purpose
+
+Candidate-only evaluation against gold-labeled corpus cases — absolute recall/precision and clean-case discipline, cheap enough to run on every meaningful pipeline change (no upstream review invocations).
+
+## Requirements
+
+### Requirement: Gold mode evaluates the candidate against labels without baseline runs
+
+The harness SHALL provide a gold-mode runner that, for a gold-labeled
+corpus case, runs `/reviso:review` against the case's pinned checkout
+(identically to parity mode's candidate leg) and judges the resulting
+findings against the case's labels file — invoking no upstream review at
+any point. Matching SHALL use the same matcher and correctness/cleanup
+tiering as parity mode.
+
+#### Scenario: No upstream invocations
+
+- **WHEN** a gold-mode run completes
+- **THEN** no `/code-review` (or other upstream review) invocation was
+  made; the only model costs are the candidate run and the judge's
+  matching calls
+
+#### Scenario: Same matcher, same calibration
+
+- **WHEN** gold mode matches candidate findings to gold labels
+- **THEN** it uses the parity harness's matcher, so one calibration
+  covers both modes
+
+### Requirement: Gold metrics separate recall, proxy precision, and clean-case discipline
+
+Gold-mode output SHALL report per-case and aggregate:
+`gold_recall_correctness` (matched correctness-tier gold issues ÷ total
+correctness-tier gold issues), a precision proxy (candidate findings
+matching any gold issue ÷ all candidate findings) explicitly labeled as a
+proxy, and — for cases marked expected-clean — the count of candidate
+findings, each of which is a false positive. Cleanup-tier gold issues the
+candidate does not match SHALL be reported informationally and SHALL NOT
+reduce gold recall.
+
+#### Scenario: Expected-clean case with findings
+
+- **WHEN** the candidate reports any finding on an expected-clean case
+- **THEN** each such finding is counted and listed as a false positive
+  for that case, with no judge call required
+
+#### Scenario: Unmatched candidate finding is not auto-penalized
+
+- **WHEN** a candidate finding matches no gold issue
+- **THEN** it lowers only the proxy-precision number and is listed for
+  potential promotion into the labels (real-but-unlabeled), mirroring
+  parity mode's claimed-wins handling
+
+#### Scenario: Cleanup-tier gold issue missed
+
+- **WHEN** a gold issue tiered cleanup has no candidate match
+- **THEN** it appears in an informational bucket and does not change
+  `gold_recall_correctness`
+
+### Requirement: Synthetic cases run against materialized throwaway repos
+
+For corpus cases marked synthetic (no upstream repo), gold mode SHALL
+materialize the case's diff into a freshly initialized throwaway git
+repository (base content committed, diff applied as the change under
+review) so the candidate reviews a real checkout; such cases SHALL be
+skipped by parity tooling.
+
+#### Scenario: Synthetic case reviewed
+
+- **WHEN** gold mode runs a synthetic case
+- **THEN** the candidate reviews a git checkout whose working diff equals
+  the fixture's diff, and the run is judged against the case's labels
+  like any other
+
+#### Scenario: Parity tooling skips synthetics
+
+- **WHEN** the parity baseline runner is pointed at a synthetic case
+- **THEN** it refuses the case, identifying it as gold-mode-only

openspec/specs/parity-eval/spec.md

@@ -105,15 +105,33 @@ correctness tier, dismissal rate <10%.
 
 The corpus format SHALL support a public tier committed under
 `eval/corpus/` and a private tier referenced by local path and never
-committed. Published eval runs (docs/evals.md) SHALL come from the public
-tier only.
+committed. Public-tier cases MAY carry a committed labels file under
+`eval/corpus/labels/<case-id>.json` in the shared label schema, with
+imported labels attributed to their source (origin, upstream license
+vendored beside the labels, import date); private-tier labels stay
+outside the repo in the same schema. Published eval runs (docs/evals.md)
+SHALL come from the public tier only, and SHALL distinguish gold-mode
+metrics (whole corpus, per release) from parity metrics (active subset,
+re-baseline cadence).
 
 #### Scenario: Private tier stays private
 
 - **WHEN** the harness runs against the Vyttle (private) corpus
 - **THEN** no corpus entry, diff content, or finding text from it lands in
   the repository
 
+#### Scenario: Imported labels carry provenance
+
+- **WHEN** a public case's labels were imported from an external benchmark
+- **THEN** the labels file names its origin and the vendored upstream
+  license covers it
+
+#### Scenario: Published metrics are not conflated
+
+- **WHEN** docs/evals.md publishes results
+- **THEN** gold-mode numbers and parity numbers are reported as distinct
+  metrics, each naming the corpus slice it covers
+
 ### Requirement: Baseline runs record comparability metadata
 
 Each baseline run SHALL record, in a metadata artifact beside its outputs:
@@ -181,3 +199,44 @@ against these labels before its parity numbers are trusted.
 - **WHEN** the calibration entry is added
 - **THEN** no diff content, finding text, or repository identity from it
   lands in this repository
+
+### Requirement: CRB import re-pins real PRs and skips unresolvable ones loudly
+
+The CRB importer SHALL derive each public corpus entry from the fixture's
+source repo and PR number by resolving current base/head SHAs via the
+GitHub API, and SHALL convert the fixture's gold issues into the shared
+label schema. A fixture whose PR or SHAs cannot be resolved SHALL be
+skipped with an explicit report line; the import SHALL never emit an
+entry with unpinned or guessed SHAs, and SHALL NOT copy upstream diff
+content into the repository.
+
+#### Scenario: Resolvable PR becomes a pinned entry
+
+- **WHEN** the importer processes a fixture whose PR still resolves
+- **THEN** the emitted entry carries the resolved base/head SHAs and a
+  labels file derived from the fixture's gold issues
+
+#### Scenario: Unresolvable PR is skipped loudly
+
+- **WHEN** a fixture's PR no longer resolves to SHAs
+- **THEN** the importer reports the skip by name and emits no entry for it
+
+### Requirement: The active parity subset is corpus data
+
+Corpus entries SHALL support an optional `active_parity` boolean; parity
+baseline tooling SHALL run only entries marked true, while gold mode
+ignores the marker. The imported corpus SHALL mark a subset (~10–15
+cases) spanning the corpus's languages and repos. (No CRB case is
+expected-clean — the task-1.1 audit found all 50 carry gold issues —
+so clean-case discipline is covered by gold mode's synthetic cases, not
+the parity subset.)
+
+#### Scenario: Parity run filters on the marker
+
+- **WHEN** a parity sweep is launched over the public corpus
+- **THEN** only `active_parity: true` entries incur baseline runs
+
+#### Scenario: Gold sweep covers everything
+
+- **WHEN** a gold-mode sweep is launched
+- **THEN** every gold-labeled entry runs, regardless of `active_parity`