Repository navigation
fix(jobs): keep the blind-turn guard's rc=7 out of bg job exit codes; map devin's non-interactive reject to permission-blocked - #37
Merged
Conversation
run_job re-enters the script under _supervise with OSRC_JOB_DIR set, and main() ran the blind-turn guard at the end of that child, so a delegate that exited 0 was recorded as failed / exit-nonzero:rc=7 whenever unrelated fleet state needed attention. Skip the guard inside a supervised job child. Also map devin's non-interactive tool-call reject (devin 3000.11 prints a warning and exits 0 instead of hanging) to permission-blocked after exit, devin lane only, tail-anchored, unless the delegate ended on OSRC::DONE. Scrubbed replays of the two affected jobs are the test fixtures.
OSRC_JOB_DIR is functional state the child legitimately reads (capture dirs) and it is inheritable: run_job exports it into the child, so a delegate that runs outsourcerer itself would have its blind-turn guard silently disabled, and delegate_codex's own error text tells users to export it. Move the exemption to --osrc-job-child-internal, consumed in main() the same way --osrc-preflight-internal is. Also: lane detection falls back to OUTSOURCERER_PROVIDER when meta.json is absent (it is jq-written, so no jq means no file and the mapping never fired), and the devin non-interactive reject maps to permission-blocked only on a clean exit 0; a nonzero exit keeps its real code on the exit-nonzero path. test_job_rc_guard_leak: fill @@DEVIN_REJECT@@ from a rot13-encoded recorded line the needle is pinned against (a wrong needle can no longer fill its own fixture); new cases pin that an inherited OSRC_JOB_DIR does not exempt, provider-only and missing meta.json still map, a nonzero exit with the reject in the tail keeps its real code, and run_job's child argv carries the sentinel. Unset OSRC_JOB_DIR and OSRC_STREAM so the suite is valid inside a job too.
Unset OUTSOURCERER_PROVIDER too (a suite run inside a devin job would inherit it and feed the provider-env fallback), carry OSRC_STREAM in the inherited-env case since a real nested delegate gets both, and pin that a non-devin provider env does not map the reject.
Owner
|
Thank you, Dani. Merged in 0.13.5. The argv sentinel instead of |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hey Alex, this one had me going for a while. Two devin jobs came back "failed" while their work was actually fine, and it turned out devin wasn't the cause.
What
_superviserecognizes devin's newer "rejected a tool call that requires confirmation ... non-interactive mode" warning and recordspermission-blocked(exit 3, reasonpermission-blocked:noninteractive-reject), notdone?. The check is devin lane only, tail-anchored, exit-0 only, and skipped if the delegate ended onOSRC::DONE.test_job_rc_guard_leak.sh, with scrubbed replays of the two jobs as fixtures, registered inconformance.sh. There's also a short note inreferences/jobs-and-safety.md.Why
On 0.13.4 two devin swe-2 bg jobs ended as
status=failed,exit=7,reason=exit-nonzero:rc=7:bg run -m swe-2, a read-only review.resultreturned a complete review (a verdict plus 8 findings), andclassifysaidREUSE-OUTPUT false-stall:deliverable.bg edit -m swe-2. The edits landed correctly. The delegate stopped when it tried to run the tests, and the log ends with devin's non-interactive warning.Devin exited 0 both times. The 7 comes from the blind-turn guard:
run_jobruns the job by re-enteringoutsourcerer.sh <verb> ...under_supervise, andmain()runs the guard at the end of that child. If anything in the fleet looked like it needed attention, the guard returned 7 and_superviserecorded it as the delegate's failure. Both out.logs end with the guard notice, printed from inside the job. (What the guard flagged in my case was my own session, fixed separately in #36. The leak holds for any flagged item, though.)On the second part: I ran devin 3000.11.1 directly. In
accept-edits, a command it wants confirmed is rejected, the warning goes to stderr, and devin exits 0. It doesn't hang like the olderchisel::repl::handler: Print mode: ...case that_printmode_needlematches. So with only the leak fixed, the edit job would have readdone?, with nothing saying the test step never ran.references/jobs-and-safety.mdalready namespermission-blockedas the state for exactly this wall.How
main(): return$_cmd_rcbefore the guard when the child carries--osrc-job-child-internal, a private argv sentinelrun_jobputs first in the child's command line andmain()consumes next to--osrc-preflight-internal. Keying onOSRC_JOB_DIRwould not work: the variable is functional state the child legitimately reads (capture dirs) and it is inheritable --run_jobexports it into the child, so a delegate that runs outsourcerer itself would have its guard silently disabled, anddelegate_codex's own error text tells users to export it. An argv sentinel cannot leak through the environment._noninteractive_reject_needleis assembled at runtime, like_printmode_needle, so reading the script can't trip it. The fixture keeps a@@DEVIN_REJECT@@placeholder that the test fills from a rot13-encoded recorded line after asserting the line equals the needle, and the doc paraphrases it, so no file in the repo carries it verbatim and a wrong needle cannot fill its own fixture.meta.json, falling back to theOUTSOURCERER_PROVIDERenvrun_jobexports (meta.json is jq-written; without jq it never exists). RespectsOSRC_NO_PRINTMODE_ABORT=1.classifyis untouched. The edit job's landed work still classifies asREUSE-OUTPUT.How to reproduce
The leak:
MAYBE STUCKorWAITING ON YOU).outsourcerer.sh bg run -m swe-2 "<any short read-only question>", then look at the job dir:statusisfailed,exitis7,reasonisexit-nonzero:rc=7, andout.logends with>>> [outsourcerer] blind-turn guard: ....The devin exit code:
Without devin:
bash plugins/outsourcerer/skills/outsourcerer/scripts/tests/test_job_rc_guard_leak.sh. Run against the previous revision's script it fails 8 checks (an inheritedOSRC_JOB_DIRdisabled the guard, the missing-meta case readdone?, a nonzero exit was rewritten to 3, and the sentinel was absent fromrun_job's argv); on this branch it's 19/19.Tests
test_job_rc_guard_leak.sh: 19/19. It covers the control (the guard still returns 7 at orchestrator level), the inherited-env non-exemption, both recorded jobs, the warning replayed on stderr, a mid-log line pushed out of the tail,OSRC::DONEwinning, a non-devin lane quoting the line, the opt-out, provider-only and missingmeta.json, a non-devin provider env, a nonzero exit with the reject in the tail, the needle not appearing verbatim in the script, andrun_job's child argv carrying the sentinel.test_blind_turn_guard,test_preflight_env_isolation,test_preflight_guard_exempt,test_job_lifecycle,test_supervise_pgroup_kill,test_autodetach,test_hardening,test_selfcontained_hardening.conformance.sh: the only failures are the ones main already has on my machine (test_autodetach,test_cline_lane,test_devin_plan_quota,test_lane_plan_limit,test_failover_pick;test_heartbeat_ownershipis flaky on main too).