Skip to content

feat(retro): add flapping detection to retro analysis skill #540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Benkapner wants to merge 7 commits into
base: main
Choose a base branch
from
+40 −0
Open

feat(retro): add flapping detection to retro analysis skill #540

Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
de40314
feat(retro): add flapping detection to retro analysis skill (#5512)
Benkapner Jul 30, 2026
ca31c01
fix(retro): address review findings on flapping detection section
Benkapner Jul 30, 2026
6495895
fix(retro): address review findings on flapping detection
Benkapner Jul 30, 2026
30c8f64
fix(retro): wire issue-URL PR lookup and add --repo to dispatch prompt
Benkapner Aug 2, 2026
b11fab1
fix(retro): address round-3 flapping detection review findings
Benkapner Aug 4, 2026
f581d10
fix(retro): address round-4 flapping detection review findings
Benkapner Aug 9, 2026
ca4b912
fix(retro): address round-5 flapping detection review findings
Benkapner Aug 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions skills/retro-analysis/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,46 @@ After subagents return their findings, use your main context to:
3. Form hypotheses about root causes
4. Decide what changes to propose and where

## Flapping detection

Check whether the workflow exhibits fix-break oscillation. Flapping wastes agent cycles and often indicates a deeper problem (conflicting instructions, flaky tests, or an approach the agent cannot converge on).

### Signals to check

Flapping detection applies to PR-based workflows with code/fix cycles. Derive the PR number from the originating URL, branching on its shape:

- If `$ORIGINATING_URL` matches `/pull/`, extract directly: `PR_NUMBER="${ORIGINATING_URL##*/}"`
- If it matches `/issues/`, check for a linked PR before skipping (issue-triggered retros routinely have downstream code dispatches once the issue reaches `ready-to-code`). Query `gh issue view "$ORIGINATING_URL" --json closedByPullRequestsReferences` and use the `repository` field on each entry to identify which repo the PR lives in. If no linked PR is found, skip flapping detection for this retro.

@waynesun09 waynesun09 Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM — No selection rule when closedByPullRequestsReferences returns multiple linked-PR entries

For issue URLs, this line says to query gh issue view "$ORIGINATING_URL" --json closedByPullRequestsReferences and "use the repository field on each entry to identify which repo the PR lives in" (plural "each entry"), but an issue can be closed by more than one PR. The text never states which entry's PR number/repo to use when multiple are returned, unlike the file's general convention of explicitly handling ambiguity elsewhere (e.g. the timestamp-window uncertainty note in the Run discovery prompt).

Suggestion: Add a tie-breaking rule — e.g. "if multiple PRs are linked, prefer the one in $REPO_FULL_NAME; otherwise use the most recently updated entry and note the ambiguity in the retro summary" or "run flapping detection against each linked PR separately."


Dispatch subagents to gather the data. Substitute `<DISPATCH_REPO>`, `<REPO>`, and `<PR_NUMBER>` with the concrete values resolved in Setup before dispatching.

@waynesun09 waynesun09 Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HIGH — <REPO> and <PR_NUMBER> substitution placeholders are never actually resolved anywhere in the file

Line 135 says: "Dispatch subagents to gather the data. Substitute <DISPATCH_REPO>, <REPO>, and <PR_NUMBER> with the concrete values resolved in Setup before dispatching." But the ### Setup block (lines 15-20) only defines ORG and DISPATCH_REPO — it never defines REPO or PR_NUMBER. PR_NUMBER is actually derived a few lines above, inside this same "Signals to check" subsection (line 132 for /pull/ URLs via PR_NUMBER="${ORIGINATING_URL##*/}"; line 133 via closedByPullRequestsReferences for /issues/ URLs) — not in Setup. Worse, <REPO> is never explicitly bound at all: for the /pull/ case there's no line equating it to $REPO_FULL_NAME, and for the /issues/ case, line 133 only says to "use the repository field on each entry to identify which repo the PR lives in" without ever instructing the agent to store that as REPO.

Every one of the three dispatch prompts (Run discovery, Review history, CI results) depends on <REPO> and <PR_NUMBER> being correctly substituted, so an implementing agent is left to guess both values — most plausibly defaulting <REPO> to $REPO_FULL_NAME even in the cross-repo issue-linked case, silently querying the wrong repo for every gh api repos/<REPO>/... call in Run discovery, Review history, and CI results.

Suggestion: Add explicit lines binding both placeholders, e.g. REPO="$REPO_FULL_NAME" when the PR is derived from a /pull/ URL; REPO="<repository.owner.login>/<repository.name>" from the matching closedByPullRequestsReferences entry when derived from an /issues/ URL. Also correct "resolved in Setup" on this line to point at this subsection instead of the unrelated ### Setup block.


- **Run discovery:** "List all code, fix, and review workflow runs via `gh run list --workflow=code.yml --repo <DISPATCH_REPO> --limit 100`, `gh run list --workflow=fix.yml --repo <DISPATCH_REPO> --limit 100`, and `gh run list --workflow=review.yml --repo <DISPATCH_REPO> --limit 100`. Filter to runs belonging to PR #<PR_NUMBER> by grepping each run's logs (`gh run view <RUN_ID> --repo <DISPATCH_REPO> --log | grep -i '<issue-or-branch-reference>'`). For each matching code/fix run, correlate it to a PR commit by matching the run's timestamp against the PR's commit history (no direct run-to-SHA mapping is exposed); if two candidate commits/runs fall within a short window, mark the correlation as uncertain. Then fetch that commit's changed files via `gh api repos/<REPO>/commits/<SHA>` (`.files`). Use workflow-run boundaries to define 'runs', not individual commits; a single run may produce more than one commit."

@waynesun09 waynesun09 Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

HIGH — Run discovery's <issue-or-branch-reference> grep target is undefined and omitted from the substitution list

The Run discovery subagent prompt filters candidate runs with gh run view <RUN_ID> --repo <DISPATCH_REPO> --log | grep -i '<issue-or-branch-reference>'. Unlike <RUN_ID> and <SHA> (legitimately left unresolved as per-iteration loop variables), <issue-or-branch-reference> is a single fixed search string the retro agent must supply once before dispatch — yet line 135's substitution instruction only lists <DISPATCH_REPO>, <REPO>, and <PR_NUMBER>, omitting it entirely, and nothing in the section states how to derive its value (e.g., from the agent/{issue}-{slug} branch convention already documented in this file's "From a PR" section, or from the issue number in $ORIGINATING_URL).

Since this grep is the sole mechanism for filtering the many candidate runs down to the ones belonging to the target PR, leaving it undefined causes the filter to either match nothing (silently reporting no flapping) or be filled in ad hoc and inconsistently by whichever agent executes it.

Suggestion: Add <issue-or-branch-reference> to the line-135 substitution list and state its source explicitly — e.g. the issue number extracted from the PR's agent/{issue}-{slug} branch name, per the "From a PR" section above.

- **Review history:** "Fetch all reviews for PR #<PR_NUMBER> via `gh api repos/<REPO>/pulls/<PR_NUMBER>/reviews --paginate`, then fetch per-review comments. Summarize the findings from each review cycle so that finding content can be compared across cycles."
- **CI results:** "For each commit correlated to a code/fix run, query `gh api repos/<REPO>/commits/<SHA>/check-runs` and report the test pass/fail results."

@waynesun09 waynesun09 Aug 12, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM — CI results subagent implicitly depends on Run discovery's output with no stated sequencing

The three subagent bullets (Run discovery, Review history, CI results) are presented as a flat bulleted list, matching the style this file otherwise uses for genuinely independent, parallel-dispatchable subagents (see "Dispatch subagents for each investigation thread", lines 108-114). But "CI results" (this line) says "For each commit correlated to a code/fix run, query ... check-runs" — the correlated-commit list only exists as an output of the "Run discovery" subagent's own work (line 137's timestamp-based correlation). Nothing in the section states that Run discovery must complete before CI results is dispatched, or that Run discovery's resulting SHA list must be fed into the CI results prompt.

Suggestion: State explicitly that Run discovery must run first and that its correlated commit SHAs should be substituted into the CI results subagent prompt, rather than presenting all three bullets as independently dispatchable.


Then check for these patterns:

1. **File oscillation:** the same file was changed in two or more consecutive runs, and the changes reverse each other (lines added in run N were removed in run N+1, or vice versa).
2. **Test result flipping:** a test that passed after run N fails after run N+1, then passes again after run N+2, and the flapping test covers a file the agent modified in the same run. Tests that flip independently of agent changes may be pre-existing flaky tests, not agent-caused oscillation.

@waynesun09 waynesun09 Aug 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MEDIUM — Test-result-flipping signal requires a test-to-file coverage mapping that no described data source provides

Pattern 2 requires knowing that "the flapping test covers a file the agent modified in the same run" to distinguish real agent-caused oscillation from pre-existing flaky tests. But the data-gathering subagent prompt only collects changed-file lists (from code/fix runs) and check-run pass/fail status per commit (via gh api repos/$REPO_FULL_NAME/commits/<sha>/check-runs) — check-runs are named CI jobs (e.g. "unit-tests", "lint"), not per-test results with file-level coverage data. No mechanism (test-name-to-file heuristic, coverage report parsing, etc.) is described anywhere for establishing that a specific flapping test actually "covers" a specific changed file, so the retro agent has no way to actually apply this exclusion criterion from the data it's instructed to collect.

Suggestion: Either specify a concrete heuristic (e.g., match test file paths whose names substring-match a changed file's basename, or parse coverage-report artifacts if one exists), or relax the pattern to something checkable from the collected data (e.g., "a test flips status across 3+ runs; treat as higher-confidence flapping if a related-by-name file also changed in the same runs") and note the strict per-file-coverage version as a future refinement once that data source exists.

3. **Cycle count:** more than 2 review-fix cycles on the same PR without convergence (the review keeps requesting changes on the same or alternating findings, e.g. a fix for one issue reintroducing a previously resolved one counts as flapping too). This threshold is a starting point; see [flapping-convergence.md](https://github.com/fullsend-ai/fullsend/blob/main/docs/problems/flapping-convergence.md) for open questions on making it configurable per repo/task type.

### When flapping is detected

Include a proposal with these specifics:

- **target_repo:** the repo where the fix should land (see Localization guidance below)
- **title:** Start with "Flapping detected:" followed by what oscillated
- **what_happened:** List each cycle with the run IDs, which files changed, and how the changes reversed
- **what_could_go_better:** Identify what might be causing the loop (conflicting review criteria, flaky test, ambiguous instructions)
- **proposed_change:** Suggest a concrete intervention (clarify the conflicting instruction, fix the flaky test, add a convergence guard)
- **validation_criteria:** Define a measurable outcome tied to the specific pattern. For example: "The next 2 fix cycles touching <file> should not re-introduce the change reverted in run N+1."

### When NOT to flag

- A single rework cycle (review requested changes, fix addressed them, review approved) is normal, not flapping.
- Different files changing across runs is normal iteration, not oscillation.
- Only flag when you see the same changes being applied and reversed repeatedly.

Comment on lines +124 to +163

@qodo-code-review qodo-code-review Bot Jul 30, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Protected skills/ file modified 📜 Skill insight § Compliance

This PR modifies a protected governance/infrastructure path (skills/retro-analysis/SKILL.md), so
it must not be auto-approved and requires explicit human review controls. Without enforcing this,
governance-critical content can change without appropriate oversight.
Agent Prompt 
## Issue description
The PR modifies a protected path (`skills/`), which must not be auto-approved and should require explicit human/CODEOWNERS review.

## Issue Context
Compliance requires raising a protected-path finding whenever files under paths like `skills/` are modified.

## Fix Focus Areas
- skills/retro-analysis/SKILL.md[124-158]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

## Before proposing: check for existing issues

**This step is mandatory.** Before including any proposal in your output, verify that no open issue already covers the same improvement. The retro agent is the primary source of systemic proposals — without this check, repeated runs produce duplicate issues that waste human triage time.
Expand Down
Loading