Repository navigation
Conversation
This was referenced Sep 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Publishes the complete compact record of unseen-validation campaign 002, including every failure, and an audit that recalculates the outcome without API calls. Stacked on #8 (the evaluation-harness branch) so the data does not obscure the grader review; after that PR reaches staging, retarget this PR to staging.
Contains all 120 result/check records, 393 drained model-call records, 120 source diffs, preregistration, outage/probe records, checksums and provenance. The offline audit verifies coverage, pass flags, model identities and the acceptance decision. All upstream errors remain scored; health probes are separate. The full 109 MB screenshot/app/trace archive is retained locally, not committed, and this limit is explicit.
Validated with
python3 evals/audit_results.py: accepted, 45/60 vs 53/60, exact recorded mean/P50 reproduced. Credentials and local path prefixes were checked before publication. These numbers apply to the frozen snapshot, not the later integration with current main. The invalidated prior campaign and future holdout requirements are documented.See the recorded result and replication guide.
GitHub CI status
Linux/macOS tests, formatting and shellcheck pass. Windows record-isolation tests already fail on the unchanged main revision (
TestTheNewestRecordComesFirst,TestNoRecordsIsNotAnError,TestAnUnwritableHomeIsSurvived): their HOME-only setup does not isolate Windows USERPROFILE. This PR reproduces those existing failures.