What happened
On PR #5851, the PR body's test plan included a checked item linking to a GitHub Actions run as evidence that postUpgradeTasks wiring worked: [x] Trigger a Renovate dry-run to verify postUpgradeTasks wiring (link). The human reviewer (waynesun09) followed the link and discovered the dry-run actually showed artifactError / No such file or directory failures — the evidence contradicted the claim. This was a chicken-and-egg problem (the dry-run ran against main which didn't have the scripts yet). The review agent ran 7 iterations across Aug 3–7 without catching this contradiction, despite the correctness sub-agent reading the PR body and having the capability to fetch GitHub API data (it fetched linked issue #5618 and used web search to verify Renovate env var naming on the final review run).
What could go better
The correctness sub-agent should verify that evidence links in the PR test plan actually support the claims being made. When the PR body contains checked test-plan items with URLs to GitHub Actions runs, the sub-agent could use gh run view or the GitHub API to check the run's conclusion and scan for error indicators. In this case, verifying the linked dry-run would have revealed the failure and prompted the author to either re-run the test or caveat that the wiring was unverified until post-merge. Confidence: high that this specific gap existed and cost a review cycle. Confidence: medium that this pattern recurs — test plan evidence links are common in infrastructure PRs but not all PRs. The correctness sub-agent already demonstrated the capability to follow links (it fetched issue #5618), so the infrastructure exists; only the instruction to verify test-plan evidence is missing.
Proposed change
In the correctness sub-agent definition (skills/pr-review/sub-agents/correctness.md in the agents repo), add guidance for the sub-agent to verify PR test-plan evidence. Specifically: when the PR body contains a test plan section with checked items that link to GitHub Actions run URLs (github.com/.../actions/runs/...), the sub-agent should fetch the run status via gh run view <run_id> --repo <repo> --json conclusion,status and check whether the conclusion is success. If the run failed or shows errors, flag it as a correctness finding (e.g., [MEDIUM] test-plan-evidence: linked CI run <URL> shows conclusion=failure, contradicting the checked test-plan claim). Scope this to GitHub Actions URLs only (not arbitrary external links) to keep cost bounded.
Validation criteria
On the next 5 PRs in fullsend-ai/fullsend that include GitHub Actions run links in their test plan section, the correctness sub-agent should either (a) verify the linked run succeeded and note it, or (b) flag a finding when the linked run's conclusion contradicts the checked claim. Validate by checking the review agent's transcript for gh run view or GitHub API calls targeting the linked run URLs.
Generated by retro agent from fullsend-ai/fullsend#5851
What happened
On PR #5851, the PR body's test plan included a checked item linking to a GitHub Actions run as evidence that postUpgradeTasks wiring worked:
[x] Trigger a Renovate dry-run to verify postUpgradeTasks wiring (link). The human reviewer (waynesun09) followed the link and discovered the dry-run actually showedartifactError/No such file or directoryfailures — the evidence contradicted the claim. This was a chicken-and-egg problem (the dry-run ran againstmainwhich didn't have the scripts yet). The review agent ran 7 iterations across Aug 3–7 without catching this contradiction, despite the correctness sub-agent reading the PR body and having the capability to fetch GitHub API data (it fetched linked issue #5618 and used web search to verify Renovate env var naming on the final review run).What could go better
The correctness sub-agent should verify that evidence links in the PR test plan actually support the claims being made. When the PR body contains checked test-plan items with URLs to GitHub Actions runs, the sub-agent could use
gh run viewor the GitHub API to check the run's conclusion and scan for error indicators. In this case, verifying the linked dry-run would have revealed the failure and prompted the author to either re-run the test or caveat that the wiring was unverified until post-merge. Confidence: high that this specific gap existed and cost a review cycle. Confidence: medium that this pattern recurs — test plan evidence links are common in infrastructure PRs but not all PRs. The correctness sub-agent already demonstrated the capability to follow links (it fetched issue #5618), so the infrastructure exists; only the instruction to verify test-plan evidence is missing.Proposed change
In the correctness sub-agent definition (
skills/pr-review/sub-agents/correctness.mdin the agents repo), add guidance for the sub-agent to verify PR test-plan evidence. Specifically: when the PR body contains a test plan section with checked items that link to GitHub Actions run URLs (github.com/.../actions/runs/...), the sub-agent should fetch the run status viagh run view <run_id> --repo <repo> --json conclusion,statusand check whether the conclusion issuccess. If the run failed or shows errors, flag it as a correctness finding (e.g.,[MEDIUM] test-plan-evidence: linked CI run <URL> shows conclusion=failure, contradicting the checked test-plan claim). Scope this to GitHub Actions URLs only (not arbitrary external links) to keep cost bounded.Validation criteria
On the next 5 PRs in fullsend-ai/fullsend that include GitHub Actions run links in their test plan section, the correctness sub-agent should either (a) verify the linked run succeeded and note it, or (b) flag a finding when the linked run's conclusion contradicts the checked claim. Validate by checking the review agent's transcript for
gh run viewor GitHub API calls targeting the linked run URLs.Generated by retro agent from fullsend-ai/fullsend#5851