Read a unified diff and a JSON object as a turn. - #364
Merged
Merged
Conversation
Traced on a 7B: asked to fix a one-line bug, it answered 'Action: patch' with a fenced unified diff, correct and complete, and heard 'patch needs Find: or Append:' ten times in a row. On another run it answered every turn as a JSON object with an action key, once with single-quoted values, and parsed to nothing ten times. Format compliance, not reasoning, which is what the literature says breaks small models. A hunk's removed lines are the Find and its added lines the Replace, context trimmed, the path from the +++ header when none is given; an addition-only hunk is an Append. A JSON object with a known action is a turn, read with json first and Python literal syntax second. An Action line still wins, and braces in prose are never a turn.
The next traced run sent the diff with no Action line at all: file headers, a hunk, the fix. Ten times, ten 'Could not parse'. A hunk with removed or added lines and a path is a patch to that file.
Contributor
|
Thanks for merging the feature to read a unified diff and a JSON object as a turn, @YauhenBichel!
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Traced on
qwen2.5-coder:7bafter #363's measurement: asked to fix a one-line bug, it answeredAction: patchwith a fenced unified diff, correct and complete, and heard "patch needs Find: or Append:" ten times in a row. On another run it answered every turn as a JSON object with anactionkey, once with single-quoted values, and parsed to nothing ten times. Format compliance, not reasoning, which is what the literature review said breaks small models.Parser
+++header when no Path is given; an addition-only hunk is an Append. Explicit fields are left alone.actionis a turn, read withjsonfirst and Python literal syntax second (single quotes). An Action line still wins, and braces in prose are never a turn.Eighteen parser tests. Full suite: 1126 tests, green.
Measured, same arm as #363 (7B, tiers 5 and 7, five passes, decision model), before and after:
REAL, +4 outside a floor of 1: said-toolow 1/5 → 4/5, fix-offbyone 0/5 → 1/5. The ledger flags that every pass scored exactly 3; with two cases mechanical at 5/5 and the gain spread over two others, that is arithmetic, not a stuck decision, but it is noted. said-offbyone and said-wrongtax are still 0/5; tracing those next.
Record:
docs/experiments/diff-json-7b.json. The write-up goes into the investigation note once #363 lands, to avoid a conflict on the same section.