Restore official DeepSearchQA grading

[ayush-or]

Reported DSQA scores use a noncanonical grader and metric

The search harness currently reduces DeepSearchQA to one judge boolean and publishes strict Fully Correct accuracy as the headline score. The paper and Kaggle starter instead require per-answer correctness decisions and rank systems by macro per-question F1. As a result, current DSQA runs are not directly comparable to the official methodology.

TL;DR

Mirror the canonical evaluator from benchmark-harness#11: restore the official prompt and free-form verdict, calculate paper precision/recall/F1 and categories, publish macro F1 as primary while retaining Fully Correct accuracy, record judge-prompt provenance, and update the trajectory viewer.

What Changed?

• Restores the versioned Kaggle/paper prompt with Gemini 2.5 Flash and per-answer correctness details.
• Calculates per-question continuous metrics and all four paper categories; macro f1_score becomes the benchmark primary score.
• Preserves strict binary accuracy as Fully Correct.
• Makes the shared judge schema optional only for DSQA; BrowseComp and WideSearch remain strict-schema judges.
• Publishes new DSQA chunks as f1_score, keeps legacy accuracy-only bundles honestly labeled, and rejects mixed old/new DSQA chunks.
• Records the versioned judge-prompt URL and SHA-256 in run provenance.
• Shows DSQA metrics, answer-part decisions, and excessive answers in the trajectory viewer.

Compatibility

Old DSQA artifacts do not contain item-level judgments, so their F1 cannot be reconstructed. They continue to publish as legacy accuracy. A run containing both legacy and macro-F1 chunks fails publication and must use a new run ID. Existing model outputs can otherwise be regraded without rerunning generation.

Validation

Credential-free validation passed:

• Harness typecheck and 250 tests
• Trajectory viewer typecheck and 7 tests
• Versioned prompt SHA-256: 9bdd0b9198244de8a78bf256b5332805d00c140e85b713f0e1878b3e4aa605a0

Paid validation used the committed run-specs/perplexity/dsqa-paper-5turn-10.toml after an approved $2.50 dry-run cap:

10 DSQA tasks, Perplexity search, 5 turns, concurrency 2
10 judged, 0 skipped, 0 parse failures
macro F1       0.774603
macro precision 0.783333
macro recall    0.781667
Fully Correct   0.600000 (6/10)
Fully Incorrect 0.100000
Partial         0.300000
actual cost     $1.437100

All 10 trajectories contained nonempty correctness-detail maps. The four non-Fully-Correct samples produced F1 values 0.4444, 0.8571, 0, and 0.4444, directly proving partial credit survives generation, Gemini grading, persistence, aggregation, publication, and reader display. No GitHub checks are configured for this repository.

Reviewer Focus

• Confirm the evaluator matches benchmark-harness#11.
• Confirm macro F1 and Fully Correct accuracy remain distinct through persistence and publication.
• Confirm legacy artifacts are never relabeled as F1.
• Confirm the viewer labels DSQA strict accuracy as Fully Correct rather than all-cells exact.

[perry-the-pr-reviewer]

Perry's Review

Verdict: 💬 Comments / questions

Note: Perry's maintainer app lacks pull_requests:write on OpenRouterTeam, so this review is posted as a comment rather than an approval. The review body recommends approval — a human maintainer can approve if they agree.

Full review

This PR restores the official DeepSearchQA grading methodology from the paper (arXiv:2601.20975v1), mirroring the canonical benchmark-harness#11. I verified the implementation against the paper's formal definitions, the reference PR, and the test suite.

Recommendation: approve. The implementation is correct, well-tested, and matches the canonical reference.

What was checked

Metric calculation (calculateDsqaGrade) — Verified precision/recall/F1 against the paper's formulas (P = |S∩G|/|S|, R = |S∩G|/|G|, F1 = harmonic mean) and all six parametric test cases. The four categorical classifications (Fully Correct, Fully Incorrect, Partially Correct, Correct with Extraneous) match the paper's disjoint set relationships. Empty correctness_details correctly fails closed — fullyCorrect requires details.length > 0, so a vacuous every(Boolean) on an empty array cannot produce a false positive.

Macro-averaging (dsqaQuestionMetrics → dsqaPrimaryScore) — Confirmed macro F1 is the mean of per-question F1 values (not pooled F1 or F1 of macro precision/recall), matching the paper's "calculated for each individual prompt and then averaged over the entire evaluation set." Epochs are mean-reduced per question before the question-level macro average. Skipped samples are excluded. The test case (5/12 macro F1) verifies the full chain.

Judge prompt and parsing (grader.ts) — The prompt SHA-256 is pinned and verified by test (9bdd0b9...). Verdict parsing handles both bare and ```json-fenced output with surrounding prose, and rejects malformed values (string booleans, non-string arrays, null). The nested "Answer Correctness" wrapper is correctly unwrapped to the flat DsqaVerdict shape.

Judge infrastructure (judge.ts) — Making jsonSchema optional is clean: when undefined, no text.format is sent (free-form output), and the body omits text entirely via conditional spread. WideSearch and answer-equivalence judges still provide schemas — the change is DSQA-only. The new test confirms body.text is undefined for unstructured specs.

Backward compatibility (publish-run.ts) — Legacy DSQA chunks (no primaryScore) continue publishing as accuracy. Mixed runs (some chunks with primaryScore, some without) are rejected at publication time. The dsqaUsesMacroF1 flag correctly gates whether macro F1 or accuracy is the primary metric and score. The samples_judged metric is summed across chunks while other metrics are weight-averaged.

Provenance (run-spec.ts) — judgePromptSource and judgePromptSha256 are recorded for DSQA, using the same sha256 function already used for inference-prompt provenance. The test verifies the expected values.

Trajectory viewer (app.js) — New dsqa_grade branch renders metrics, correctness details, and excessive answers. The legacy all_expected_answers_found block is intentionally retained for backward compatibility with old trajectories. The exactLabel logic correctly maps "macro F1" → "fully correct" and "item F1" → "all cells exact".

One suggestion

The judgeCall docstring should be updated to note that the strict-json_schema guarantee only applies when jsonSchema is provided — DSQA now uses free-form output. See inline comment.

Risk assessment

Dimension	Severity	Risk	Reasoning
Implementation risk	🟩	Low	Well-tested, matches the canonical reference PR and the paper's formal definitions
Premise risk	🟩	Low	The macro-F1 methodology is directly from the paper; the prompt is pinned by SHA-256
Estimated impact	🟩	Low	Worst case is incorrect DSQA scores, but backward-compatible, detectable via tests, and cheap to re-run

Risk Factor	Severity	Risk	Reasoning
Reversibility	🟩	Low	Reverting restores old accuracy-only behavior
Detectability	🟩	Low	250 harness tests + 7 viewer tests + paid validation run all pass
Blast radius	🟩	Low	Only affects DSQA suite scoring; BrowseComp and WideSearch unchanged
Data integrity		None	Legacy chunks preserved as-is; no persisted data is mutated
Financial exposure		None	No billing or payment surface
Security and privacy exposure		None	No credentials or PII involved
Propagation	🟩	Low	Only published DSQA scores carry forward
Availability		None	No serving path affected
Recovery cost	🟩	Low	Re-run the benchmark with corrected code
Time to correct	🟩	Low	Fix and re-run; no coordination needed

[perry-the-pr-reviewer]

Now that jsonSchema is optional, the docstring above (Strict \json_schema` output means the model cannot emit malformed JSON`) only holds for specs that provide a schema. DSQA uses free-form output and can emit unparseable text, making a parse failure more likely than the comment implies.

▶ Suggestion: Update the docstring to clarify that the malformed-JSON guarantee applies only when jsonSchema is provided, and that free-form specs (e.g. DSQA) rely on parseVerdict to handle prose-wrapped or fenced JSON.

[ayush-or]

Superseded by #6, which targets main directly, includes this DSQA repair through merged benchmark-harness #11, and replaces the duplicated package with the exact upstream subtree.