Skip to content

fix(eval): reconcile LongMemEval resume no-ops at zero cost - #5074

Open
Masashi-Ono0611 wants to merge 1 commit into
garrytan:masterfrom
Masashi-Ono0611:feat/longmemeval-resume-noop-zerocost
Open

fix(eval): reconcile LongMemEval resume no-ops at zero cost#5074
Masashi-Ono0611 wants to merge 1 commit into
garrytan:masterfrom
Masashi-Ono0611:feat/longmemeval-resume-noop-zerocost

Conversation

@Masashi-Ono0611

Copy link
Copy Markdown
Contributor

Problem

gbrain eval longmemeval --resume-from <file> has a resume-noop path: 0 remaining questions to answer and 0 judge-backfill calls needed. This path calls no provider (no reader, no embedder, no judge) — actual spend is $0 — but it never wrote to GBRAIN_EVAL_ACTUAL_COST_FILE, so scripts/eval-spend-guard.sh had no actual-cost evidence for this invocation and always fell back to the (nonzero) pre-run estimate for it. Local TODO context: TODOS.md's existing "P3 — gbrain eval longmemeval writes the spend guard's actual-cost file" entry.

Fix (narrow, invocation-scoped only)

  • src/commands/eval-longmemeval.ts: on the resume-noop branch only (0 questions, 0 backfill), write a canonical v1 JSON receipt ({"schema_version":1,"scope":"invocation","complete":true,"reason":"resume_noop","cost_usd":0}) to GBRAIN_EVAL_ACTUAL_COST_FILE (or the actualCostFile test seam) via a temp-file-then-atomic-rename (wx flag, 0o600), before finishRun (whose quality gate may still exit nonzero independently of this invocation's zero spend).
  • scripts/eval-spend-guard.sh: accepts $0 reconciliation only for this exact canonical shape — fixed key order, exact key set, schema_version:1, scope:"invocation", complete:true, reason:"resume_noop", cost_usd:0 — matched via a regex against the whole file content (not stripped-then-matched, to reject NUL-byte smuggling: an LC_ALL=C tr -d '\000' round-trip is byte-compared against the original before accepting). Every other shape (missing/duplicate/extra fields, partial receipt, bare 0, legacy {"cost_usd":0}, wrong field types, whitespace-only differences inside string values) falls back to the pre-run estimate, unchanged from existing behavior.
  • Fail-closed freshness guard: the guard only accepts a zero receipt when the target path did not already exist (or exist as a symlink) at guard start, checked before the child process runs. This stops a stale zero-receipt left over from a prior run at the same explicit path from being credited to a new invocation that itself did no work-relevant reconciliation. The guard's own default (auto-generated) temp path always satisfies this; an operator who explicitly sets GBRAIN_EVAL_ACTUAL_COST_FILE needs a fresh path per invocation for the zero-cost exception (documented). This freshness requirement applies only to the new zero-receipt path — existing positive-cost-file handling is unchanged.
  • No changes to the normal (non-resume-noop) evaluation path's cost accounting, spend.posture handling, or pricing tables. Historical judge_cost_usd / cumulative QA totals are never copied into the zero receipt.
  • docs/operations/spend-controls.md: new subsection documenting the receipt shape and the fail-closed conditions above.

Tests

test/eval-spend-guard.test.ts (+3 new tests) and test/eval-longmemeval-judge.slow.test.ts (assertions added to 2 existing tests):

  • Canonical receipt accepted in 2 formats (compact, pretty-printed with indentation/newlines) and 13 near-miss variants rejected (wrong complete/schema_version/scope/reason value, string-typed cost_usd, truncated JSON, extra whitespace token, missing field, duplicate key, bare 0, legacy {"cost_usd":0}) — all correctly falling back to the estimate.
  • NUL-byte smuggling attempt (a \0 inside a string value that would vanish under naive whitespace-stripping) is rejected, not silently normalized away.
  • A pre-existing zero receipt at the target path is never credited to a subsequent invocation that reserves the same path (freshness check).
  • Live integration: a --judge --resume-from backfill call that needs 0 further reader/judge calls now writes the zero receipt and the guard's ledger records cost_usd: 0 for it; a separate live judge run (nonzero real work) does NOT write a zero receipt, confirmed by existsSync(...) being false on that path.

bun test test/eval-spend-guard.test.ts test/eval-longmemeval-judge.slow.test.ts: 44 pass, 0 fail, 524 expect() calls (reproduced independently by the reviewer outside the implementer's original sandboxed worktree). bun run typecheck and bun run verify (54 checks) both clean, reproduced independently. git diff --check clean. All provider calls in the test suite are existing mocks — no real API calls, no billing impact, confirmed by reading the diff before submission.

What I could not verify

No new coverage for a resume-noop scenario where GBRAIN_EVAL_ACTUAL_COST_FILE is unset entirely (the write is skipped via the existing if (costFile) guard) beyond the pre-existing default-temp-path tests already covering that shape indirectly.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Kthq1tdcmzggtc8if3jHtW

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant