bench: junit-grade testFileFrom=task keeps grading assets out of the workDir - #110
Open
lec77 wants to merge 1 commit into
Open
bench: junit-grade testFileFrom=task keeps grading assets out of the workDir#110lec77 wants to merge 1 commit into
lec77 wants to merge 1 commit into
Conversation
…workDir A junit-graded task had to ship its test file through fixtures/, which is copied into the agent-visible workDir — so the grading assets, including expected and golden outputs, sat in the directory the agent is free to read and write. An agent that reads the answer key is not being graded on the task. Add an optional testFileFrom on the payload: with "task", testFile and its sibling expected/golden files resolve against the task directory instead of fixtures/, so they never enter the workDir. The test subprocess still runs with cwd=workDir, so nothing changes for the code under test. Bench conditions forward task.taskDir through EvaluatorConfig automatically, and the default stays fixtures/ so existing tasks are unaffected. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A
junit-gradetask can only ship its test file throughfixtures/, andfixtures/is copied into the workDir the agent runs in. So the grading assets — the test file itself, plus any siblingexpected_*/ golden outputs it reads — sit in a directory the agent can freely list, read, and overwrite. An agent that reads the answer key is not being graded on the task, and nothing in the results distinguishes that run from an honest one.Change
Adds an optional
testFileFromto the evaluator payload:"fixtures"(default) — today's behaviour, unchanged for every existing task."task"—testFileand its sibling expected/golden files resolve against the task directory, which is never copied into the workDir.The test subprocess still runs with
cwd = workDir, so nothing changes for the code under test — only where the grader loads its own assets from. Bench conditions forwardtask.taskDirthroughEvaluatorConfigautomatically, so a task opting in needs no wiring.Verification
bunx tsc --noEmitclean;bun test1299 pass / 0 fail.test/bench/evaluators/junit-grade.test.tscovers both resolution modes, sibling-file resolution, and the missing-taskDir error path;test/bench/run-condition.test.tsasserts conditions forwardtaskDir.🤖 Generated with Claude Code