What happened
PR #766 originally changed only the policies/fix.yaml header comment — replacing "Identical to the code agent policy" with wording allowing intentional divergence — without adding the missing registry.yarnpkg.com endpoint or npx/yarn/yarnpkg/pnpm binaries that open issue #532 had identified as a High-severity gap. The fullsend-ai-review[bot] ran on commit 4bf08a2 (run 31601835105) and flagged a [low] scope-intent-misalignment, noting the comment "softens the synchronization commitment from #532" but accepting it as "a reasonable architectural position." A manual review sweep (waynesun09) caught this as HIGH: the PR was reframing a known functional gap as intentional design rather than fixing it. This finding drove the PR to add the missing entries and ultimately close #532.
What could go better
The bot's intent-coherence sub-agent saw the #532 reference and noticed the tension, but rated it LOW because it accepted the PR author's re-framing narrative without checking whether the underlying issue's requirements were met. The triage summary on #532 enumerated 5 specific missing entries, and the diff at commit 4bf08a2 addressed zero of them. The sub-agent should have fetched the linked issue's triage summary, compared it against the diff's substantive changes (not comments), and escalated to HIGH when the diff only modified documentation while the issue's concrete requirements remained unaddressed.
Confidence: High. The bot's own LOW finding shows it had the right signal — the failure was in assessment, not detection. The specific anti-pattern (doc/comment-only changes that re-frame an open bug as by-design) is a distinct failure mode from general AC-vs-implementation mismatches covered by #725 and #713. Those issues target broad implementation-vs-AC verification; this targets the narrower pattern of non-substantive resolution through narrative reframing.
Proposed change
Add a detection step to the intent-coherence sub-agent definition (skills/pr-review/sub-agents/intent-coherence.md) for the following pattern:
- When: The PR diff modifies files that have associated open issues (referenced in PR body, commit messages, or GitHub's linked-issues sidebar), AND the diff to those files contains only comment, documentation, or metadata changes (no substantive configuration or code changes).
- Then: Fetch the linked issue body and any triage summary via
gh api repos/{owner}/{repo}/issues/{number}. Extract the issue's stated requirements (missing entries, expected behavior changes, acceptance criteria).
- Assess: If the issue identifies specific concrete gaps (missing config entries, broken behavior, code bugs) and the diff does not address any of them, raise a
[high] non-substantive-resolution finding explaining that the PR re-frames the issue through documentation changes rather than implementing the fix. Include the specific unaddressed requirements from the issue.
- Do not flag when: the PR body explicitly states it is a partial fix or preparatory step (e.g., "Phase 1 of #N"), or the linked issue is labeled as a documentation-only request.
This should cross-reference issues #725 and #713 — if either is implemented first, this detection may be subsumed by the broader AC verification. But the narrow pattern (doc-only reframing) is worth targeting independently because it has a high true-positive rate and a clear signal (zero substantive lines changed vs. concrete issue requirements).
Validation criteria
Construct a test case modeled on PR #766's initial state: a PR that modifies only comments in a policy file while a linked open issue identifies specific missing configuration entries. The intent-coherence sub-agent should produce a HIGH finding identifying the unaddressed requirements. Validate against 3 scenarios: (1) comment-only change with open linked issue listing concrete gaps → HIGH finding expected; (2) comment change accompanied by the substantive fix → no non-substantive-resolution finding expected; (3) comment-only change with no linked issue → no finding expected (no issue to cross-reference). The next 3 review agent runs on PRs in this repo that reference open issues should demonstrate the sub-agent cross-referencing linked issue requirements.
Generated by retro agent from #766
What happened
PR #766 originally changed only the
policies/fix.yamlheader comment — replacing "Identical to the code agent policy" with wording allowing intentional divergence — without adding the missingregistry.yarnpkg.comendpoint ornpx/yarn/yarnpkg/pnpmbinaries that open issue #532 had identified as a High-severity gap. Thefullsend-ai-review[bot]ran on commit4bf08a2(run 31601835105) and flagged a[low]scope-intent-misalignment, noting the comment "softens the synchronization commitment from #532" but accepting it as "a reasonable architectural position." A manual review sweep (waynesun09) caught this as HIGH: the PR was reframing a known functional gap as intentional design rather than fixing it. This finding drove the PR to add the missing entries and ultimately close #532.What could go better
The bot's intent-coherence sub-agent saw the #532 reference and noticed the tension, but rated it LOW because it accepted the PR author's re-framing narrative without checking whether the underlying issue's requirements were met. The triage summary on #532 enumerated 5 specific missing entries, and the diff at commit
4bf08a2addressed zero of them. The sub-agent should have fetched the linked issue's triage summary, compared it against the diff's substantive changes (not comments), and escalated to HIGH when the diff only modified documentation while the issue's concrete requirements remained unaddressed.Confidence: High. The bot's own LOW finding shows it had the right signal — the failure was in assessment, not detection. The specific anti-pattern (doc/comment-only changes that re-frame an open bug as by-design) is a distinct failure mode from general AC-vs-implementation mismatches covered by #725 and #713. Those issues target broad implementation-vs-AC verification; this targets the narrower pattern of non-substantive resolution through narrative reframing.
Proposed change
Add a detection step to the intent-coherence sub-agent definition (
skills/pr-review/sub-agents/intent-coherence.md) for the following pattern:gh api repos/{owner}/{repo}/issues/{number}. Extract the issue's stated requirements (missing entries, expected behavior changes, acceptance criteria).[high] non-substantive-resolutionfinding explaining that the PR re-frames the issue through documentation changes rather than implementing the fix. Include the specific unaddressed requirements from the issue.This should cross-reference issues #725 and #713 — if either is implemented first, this detection may be subsumed by the broader AC verification. But the narrow pattern (doc-only reframing) is worth targeting independently because it has a high true-positive rate and a clear signal (zero substantive lines changed vs. concrete issue requirements).
Validation criteria
Construct a test case modeled on PR #766's initial state: a PR that modifies only comments in a policy file while a linked open issue identifies specific missing configuration entries. The intent-coherence sub-agent should produce a HIGH finding identifying the unaddressed requirements. Validate against 3 scenarios: (1) comment-only change with open linked issue listing concrete gaps → HIGH finding expected; (2) comment change accompanied by the substantive fix → no non-substantive-resolution finding expected; (3) comment-only change with no linked issue → no finding expected (no issue to cross-reference). The next 3 review agent runs on PRs in this repo that reference open issues should demonstrate the sub-agent cross-referencing linked issue requirements.
Generated by retro agent from #766