Skip to content

fix(cycle): track bracket depth in extract-atoms trailing-prose recovery - #5064

Open
Masashi-Ono0611 wants to merge 1 commit into
garrytan:masterfrom
Masashi-Ono0611:fix/extract-atoms-trailing-prose-bracket-depth
Open

Masashi-Ono0611 wants to merge 1 commit into
garrytan:masterfrom
Masashi-Ono0611:fix/extract-atoms-trailing-prose-bracket-depth

Conversation

@Masashi-Ono0611

Copy link
Copy Markdown
Contributor

What

parseArrayAtOffset (the atoms-extraction JSON parser in src/core/cycle/extract-atoms.ts) recovers a valid atoms array from a response that has trailing prose after it by trimming back to slice.lastIndexOf(']') when a whole-slice JSON.parse fails. If the trailing prose itself contains a bracketed citation (e.g. [Source: X] — a shape this brain's own house style encourages, per docs/architecture/... extraction prompts), lastIndexOf(']') finds the citation's closing bracket instead of the array's own, so the recovery slice spans past the real array into the dangling citation text and fails to parse. A well-formed atoms array is then reported as unparseable JSON array, which counts as a deterministic failure toward a page's tombstone threshold.

Example:

[{"title":"t","atom_type":"insight","body":"b"}]
See [Source: alice-example, agent session, 2026-09-03].

Before this fix: unparseable JSON array (the citation's ] hijacks the trim boundary). After: parses correctly, 1 atom recovered.

Fix

Added findArrayCloseIndex(), which walks the string from the array's own opening [, tracking bracket depth and skipping [/] characters that appear inside JSON string literals (honoring \" escapes). This finds the array's own true closing bracket regardless of what brackets appear afterward in trailing prose. parseArrayAtOffset now uses this instead of the naive lastIndexOf(']').

Scope is limited to this one recovery path inside parseArrayAtOffset: no exported signature changed, no new CLI/config surface, no other parser touched.

One side effect worth calling out explicitly (see the NOTE comment added on parseArrayAtOffset): because this now finds each candidate's own array boundary correctly instead of occasionally over-running into later text, a response embedding two complete top-level atoms arrays (e.g. a "draft, then corrected answer" pattern — not a documented or prompted output shape) will now have the FIRST array win, where before it sometimes (accidentally, via the same bug) fell through to the last one. This is the anchor scan's existing, already-documented "first candidate that parses to >=1 atom wins" policy applying correctly, not a new precedence rule. A test pins this explicitly (two complete top-level arrays: the FIRST one wins...).

Testing

  • Added regression tests to the existing test/extract-atoms-array-anchor-scan.test.ts (created for a prior related anchor-scan fix): the citation-hijack repro, an unmatched-trailing-bracket variant, the two-full-arrays precedence pin, and a test exercising escaped quotes / bracket-like substrings inside strings / a nested array on the recovery path together.
  • All 4 existing tests in files touching this parser (test/extract-atoms-array-anchor-scan.test.ts, test/llm-json-reasoning-ladder.test.ts, test/cycle/extract-atoms-synthesize-concepts.test.ts, test/extract-atoms-failure-classes.test.ts) still pass unmodified in behavior — 101 tests pass across those 4 files.
  • bun run typecheck clean.
  • bun run verify (54 checks) passes.
  • scripts/module-size-limits.tsv ceiling for this file bumped 1742 -> 1788 (mechanical, same-commit, per the repo's module-size ratchet rule) to cover the added helper + doc comments.

Discrimination test: reverted src/core/cycle/extract-atoms.ts to merge-base, ran test/extract-atoms-array-anchor-scan.test.ts → 19 pass / 4 fail. Restored → all pass.

Not verified

  • No live LLM was run against this; the repro is a synthetic string constructed to match the documented failure mode (the same technique the existing test/extract-atoms-array-anchor-scan.test.ts file already uses for its other cases).

parseArrayAtOffset's trailing-prose recovery used slice.lastIndexOf(']')
to find the array's closing bracket when the whole-slice JSON.parse
failed. A valid atoms array followed by a bracketed citation (e.g.
"...]\nSee [Source: X]." -- a shape gbrain's own house style encourages)
has that citation's ']' sort after the array's real terminator, so the
naive scan trimmed back to the wrong bracket and included the dangling
citation text in the re-parse, which then failed as "unparseable JSON
array" even though the array itself was well-formed.

Replace the naive scan with findArrayCloseIndex(), which tracks bracket
depth from the array's opening '[' and skips brackets inside JSON string
literals (honoring \" escapes), so it lands on the array's true closing
bracket regardless of what brackets appear in trailing prose.

Scope is limited to parseArrayAtOffset's internal recovery logic; the
function signature and callers (parseAtomsOutcomeInner, the anchor scan)
are unchanged. Bumped extract-atoms.ts's module-size-limits.tsv ceiling
(1742 -> 1780) for the added helper, per the module-size ratchet rule.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Kthq1tdcmzggtc8if3jHtW
@Masashi-Ono0611
Masashi-Ono0611 marked this pull request as ready for review September 13, 2026 13:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant