fix(eval): reconcile LongMemEval resume no-ops at zero cost - #5074
Open
Masashi-Ono0611 wants to merge 1 commit into
Open
fix(eval): reconcile LongMemEval resume no-ops at zero cost#5074Masashi-Ono0611 wants to merge 1 commit into
Masashi-Ono0611 wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
gbrain eval longmemeval --resume-from <file>has a resume-noop path: 0 remaining questions to answer and 0 judge-backfill calls needed. This path calls no provider (no reader, no embedder, no judge) — actual spend is $0 — but it never wrote toGBRAIN_EVAL_ACTUAL_COST_FILE, soscripts/eval-spend-guard.shhad no actual-cost evidence for this invocation and always fell back to the (nonzero) pre-run estimate for it. Local TODO context: TODOS.md's existing "P3 —gbrain eval longmemevalwrites the spend guard's actual-cost file" entry.Fix (narrow, invocation-scoped only)
src/commands/eval-longmemeval.ts: on the resume-noop branch only (0 questions, 0 backfill), write a canonical v1 JSON receipt ({"schema_version":1,"scope":"invocation","complete":true,"reason":"resume_noop","cost_usd":0}) toGBRAIN_EVAL_ACTUAL_COST_FILE(or theactualCostFiletest seam) via a temp-file-then-atomic-rename (wxflag,0o600), beforefinishRun(whose quality gate may still exit nonzero independently of this invocation's zero spend).scripts/eval-spend-guard.sh: accepts $0 reconciliation only for this exact canonical shape — fixed key order, exact key set,schema_version:1,scope:"invocation",complete:true,reason:"resume_noop",cost_usd:0— matched via a regex against the whole file content (not stripped-then-matched, to reject NUL-byte smuggling: anLC_ALL=C tr -d '\000'round-trip is byte-compared against the original before accepting). Every other shape (missing/duplicate/extra fields, partial receipt, bare0, legacy{"cost_usd":0}, wrong field types, whitespace-only differences inside string values) falls back to the pre-run estimate, unchanged from existing behavior.GBRAIN_EVAL_ACTUAL_COST_FILEneeds a fresh path per invocation for the zero-cost exception (documented). This freshness requirement applies only to the new zero-receipt path — existing positive-cost-file handling is unchanged.spend.posturehandling, or pricing tables. Historicaljudge_cost_usd/ cumulative QA totals are never copied into the zero receipt.docs/operations/spend-controls.md: new subsection documenting the receipt shape and the fail-closed conditions above.Tests
test/eval-spend-guard.test.ts(+3 new tests) andtest/eval-longmemeval-judge.slow.test.ts(assertions added to 2 existing tests):complete/schema_version/scope/reasonvalue, string-typedcost_usd, truncated JSON, extra whitespace token, missing field, duplicate key, bare0, legacy{"cost_usd":0}) — all correctly falling back to the estimate.\0inside a string value that would vanish under naive whitespace-stripping) is rejected, not silently normalized away.--judge --resume-frombackfill call that needs 0 further reader/judge calls now writes the zero receipt and the guard's ledger recordscost_usd: 0for it; a separate live judge run (nonzero real work) does NOT write a zero receipt, confirmed byexistsSync(...)being false on that path.bun test test/eval-spend-guard.test.ts test/eval-longmemeval-judge.slow.test.ts: 44 pass, 0 fail, 524 expect() calls (reproduced independently by the reviewer outside the implementer's original sandboxed worktree).bun run typecheckandbun run verify(54 checks) both clean, reproduced independently.git diff --checkclean. All provider calls in the test suite are existing mocks — no real API calls, no billing impact, confirmed by reading the diff before submission.What I could not verify
No new coverage for a resume-noop scenario where
GBRAIN_EVAL_ACTUAL_COST_FILEis unset entirely (the write is skipped via the existingif (costFile)guard) beyond the pre-existing default-temp-path tests already covering that shape indirectly.🤖 Generated with Claude Code
https://claude.ai/code/session_01Kthq1tdcmzggtc8if3jHtW