autonomy-readiness

Gathers observations from fullsend-ai/fullsend and fullsend-ai/agents about how the fullsend review agent performs relative to human reviewers, and turns the gaps into concrete recommendations — so we can start trusting the review agent to do what a human would do, or to correctly flag when it's unsure.

This isn't starting from zero

fullsend already dogfoods itself: every PR in fullsend-ai/fullsend and fullsend-ai/agents gets reviewed by fullsend-ai-review[bot], and on close/merge a fullsend-ai-retro[bot] runs a retrospective that includes a skill literally called autonomy-readiness (fullsend-ai/agents skills/autonomy-readiness/SKILL.md). That skill already computes, per PR, the delta between what the review agent found and what human reviewers found, classifies gaps, and proposes fixes.

So the single-PR analysis this project might otherwise have had to build already runs in production, automatically, on every PR. What doesn't exist yet is the rollup layer: turning hundreds of individual per-PR retrospectives (plus the ~18 months of GitHub issues they generated) into a periodic, quantified answer to "where are we actually closing the gap, and where are we not."

There's also a validity question worth taking seriously: the retro agent is grading the review agent's homework using the same kind of model doing the grading. An independent, second-opinion check on a sample of PRs is part of this project's job, not just aggregation.

Baseline findings (from the existing issue tracker)

Two distinct evidenced clusters dominate agent/review-labeled issues in fullsend-ai/fullsend as of 2026-08:

1. Redundant re-dispatch (#2587 + #2599, 75 evidence issues combined) — the same PR gets re-reviewed dozens of times on no-op/rebase commits, burning cost with zero new findings. This is a cost/efficiency gap, not a review-quality gap.
2. Verdict-severity mismatch (#2940, 17 evidence issues) — the agent posts a real medium/high-severity finding but the PR review verdict comes back COMMENT or APPROVE instead of CHANGES_REQUESTED, so the finding merges unaddressed. This is the dominant review-quality gap — the kind the user actually asked about — and it's notable because the review pipeline already has a self-consistency check meant to catch exactly this (blocking language in the summary should force escalation to request-changes). Either that check is failing in practice, or findings are being mis-classified as "advisory" when they describe an actual functional bug. That's a good first thing for the audit skill (below) to investigate directly.

Other smaller but real clusters: reviewing draft PRs (#1715, 9), re-raising findings the author already dismissed (#1672, 6), no verdict for closing policy-violating bot PRs (#1980, 6), diminishing-value post-approval findings (#3025, 3).

Data collection wrinkle: issues titled "Evidence for #N" stopped being filed around 2026-07-20 — fullsend-ai/agents skills/retro-analysis/SKILL.md now explicitly forbids filing them (they were noise; the fix landed in fullsend-ai/agents PR #85). Evidence discovered after that date lives only inside free-form retro PR-comment summaries, never as a new issue. Any gathering tool has to read both eras: historical evidence via issue search, recent evidence via PR comment text.

Architecture: three skills

1. gather-review-observations — script-driven. Given a time period, pulls every PR touched by the review/retro agents across both repos and produces structured per-PR JSON: review-agent findings, retro summary (if any), and human review activity, cleanly separated from bot noise (Qodo, Codecov, github-actions).

2. audit-pr-review-gap — launched as a sub-agent (to keep the main context free for cross-PR synthesis). Given one PR's gathered data, independently computes the agent-vs-human delta — mirroring the logic in fullsend-ai/agents skills/autonomy-readiness/SKILL.md so output is comparable — and separately checks whether the retro agent's own self-assessment (when one exists) agrees with that independent read. This is the second-opinion check.

3. mine-review-gap-patterns — cross-PR aggregation. Clusters gap findings from many audit-pr-review-gap runs, cross-references each cluster against the known agent/review issue catalog to tell "more evidence for a known gap" apart from "a genuinely new pattern," and reports frequency/trend over the period.

See PLAN.md for the detailed design, data schemas, and rollout phases.

Data sources & parsing rules

• Review agent comments: author is the bot's GitHub App identity and body contains <!-- fullsend:review-agent -->. Ignore <!-- fullsend:agent-status:... --> — that's a transient lifecycle ping, not a finding.
• Retro comments: identified by author only (fullsend-ai-retro[bot]) — there is no marker on them. Content is free-form markdown (the .summary field of schemas/retro-result.schema.json in fullsend-ai/agents); no guaranteed heading structure, so parsing needs an LLM read, not regex.
• Human signal: gh api .../pulls/{n}/reviews and .../issues/{n}/comments, filtered to user.type == "User".
• Coverage caveat: outcome labels (ready-for-merge, requires-manual-review, rejected) are applied for approve/comment/reject verdicts but not for request-changes — don't use labels alone to decide whether a PR was reviewed; check for the marker comment.
• Repos in scope: fullsend-ai/fullsend, fullsend-ai/agents.

Status

Planning stage. No code yet — see PLAN.md for the build plan.