Skip to content

fix(analyzer): avoid false shell parse limits in Markdown - #627

Open
mohgupta-ship-it wants to merge 25 commits into
mainfrom
codex/fix-prose-shell-parser-context
Open

mohgupta-ship-it wants to merge 25 commits into
mainfrom
codex/fix-prose-shell-parser-context

Conversation

@mohgupta-ship-it

@mohgupta-ship-it mohgupta-ship-it commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Powered by Codex, on behalf of the repository contributor.

Benign referenced Markdown guides could produce a false static_parse_limit, 50% coverage, HIGH AE1, score 25 / CAUTION, and strict exit 1. Reproductions include ordinary negative contractions in prose and list items, and layout words supplying flag-like evidence to an unrelated malformed span.

This change gives a narrow class of complete sentences contextual apostrophe ownership, including already parsed list-item bodies, and confines exhausted-span evidence to its current command. Existing code/container boundaries, enclosing shell ownership, parser limits, cancellation, raw threat detection, and downstream ledger/AE1 behavior remain enforced.

Validation on ab81c519f10d7498214b8a0ccd528d928437ec86:

  • 123 focused regression tests and 106 adjacent Markdown tests pass.
  • Seven unchanged synthetic contracts pass through the production CLI. The formerly failing bullet guide now has complete coverage, score 0 / SAFE, no findings, strict exit 0, and correct reference accounting. The runtime-selected command control remains incomplete.
  • Controls cover unordered, ordered and nested lists, continuations, padding, CRLF, Unicode source offsets, code/fences, enclosing quotes, runtime helpers, repeated source occurrences, deduplication and scoring parity, and cancellation.
  • Repository CI-scope lint, formatting, diff and signoff checks pass. Independent spec, security and runtime reviews, followed by an evidence-bound judge, found no unresolved code findings.

Fresh hosted CI and separately scheduled full/combined replay remain validation gates. Earlier full-suite/stress results are not presented as measurements of this new commit. The sentence grammar remains deliberately narrow; broader Unicode sentences and conservative quoted/deeply indented contexts are outside this extension.

Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
Signed-off-by: Mohit Gupta <mohgupta@nvidia.com>
@mohgupta-ship-it
mohgupta-ship-it marked this pull request as ready for review September 24, 2026 14:01

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 92f1124181a574d28fe88f775a1d7c1346ffe6c7 against base 5cbb8a4d02e8431ce0a075aa2b417ea3d912871a. No actionable defects found in this review.

Checked the sentence grammar and parsed-list context, enclosing shell/code ownership, source-offset preservation, bounded exhausted-command evidence, cancellation, and downstream ledger/CLI behavior. Also reviewed the refinement's effect outside Markdown through shell, Perl, and PowerShell controls.

Validation: 1,799 focused tests passed: 123 new prose-context regressions, 186 adjacent Markdown ownership/table tests, and 1,490 shell/reconstruction/embedded-language controls. Independent base/head probes confirmed that benign prose and bullet contractions change from static_parse_limit to complete coverage; a subsequent rm -rf / retains TM1, and runtime-selected commands plus enclosing bash -c strings remain partial.

This review does not claim a full-suite or live-provider run.

At review time, CI for this exact head reports action_required; successful repository CI is still required before merge. This review does not constitute combined 2.12.1 release qualification.

Review performed with Codex on behalf of Narendran Raghavan.

@rng1995 rng1995 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Completed a fresh second pass at 8a7ee52c07b3c6d279b02e755834dc6dea92a9d0. I found no actionable defect introduced by this change and approve the source change.

I independently traced apostrophe eligibility, shell-word ownership, cancellation, and the exhausted-command refinement, including its use outside Markdown. Fresh base/head probes covered 720 command/layout combinations across dynamic executables, nested substitutions, shell wrappers, code spans, fences, lists, quotes, continued lines and operands beyond the tokenizer bound. On the head, all 720 contraction/no-contraction control pairs have matching completeness behavior. An additional 21 enclosing-shell ownership cases retain their apostrophes and source length; 20 grammar/whitespace/repetition boundary cases were checked. Existing unsupported shapes were compared with the base and were not attributed to this PR.

These independent checks supplement the 1,799 focused tests from the first pass. After main was merged during this review, I verified both PR-specific files are byte-for-byte unchanged and reran 150 prose-context/shared-runner tests plus all 720 paired controls on this new head; all passed. Repository CI and combined-release qualification remain separate requirements. This approval covers the commit above and does not claim a full-suite or live-provider run.

Review performed with Codex on behalf of Narendran Raghavan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants