What happened
On PR #731, the review agent ran 3 times (run 31424224434, run 31501630595, run 31638890823). The PR added AGENTS.md guidance summarizing ADR 0080 and ADR 0081 rules about harness env var literal defaults. The review agent's correctness sub-agent verified that stated claims were technically sound — confirming harness env var literal semantics, base: composition overrides, and ${VAR} expansion behavior against actual harness files. However, all 3 runs produced only a protected-path governance finding. Human reviewer waynesun09 then identified two substantive gaps by reading the cited ADRs: (1) the guidance omitted ADR 0081's explicit CI-runtime exception (values derived from github.event.*, build matrix variables, or non-static secrets may live in the workflow env: block), and (2) the rule was not scoped to distinguish static tuning defaults from per-run computed passthrough values like CODE_ALLOWED_TARGET_BRANCHES. Both findings required comparing the documentation's summary against the full text of the referenced ADRs.
What could go better
The correctness sub-agent's current guidance includes 'design document alignment — if the plan references a design document or ADR, are the claims consistent with the referenced source.' This check verifies accuracy (are stated claims correct?) but not completeness (does the summary cover all significant provisions?). On this PR, the stated claims were accurate — the review agent confirmed this. The gap was that the documentation omitted two significant ADR provisions entirely. The correctness sub-agent did not read ADR 0080 or ADR 0081 to check whether the AGENTS.md summary faithfully covered all their key rules, exceptions, and scoping.
Confidence is high: the review agent's own comment states 'the correctness sub-agent verified all claims,' confirming the accuracy check ran but completeness was not evaluated. Both human findings (ADR 0081 exception, scope distinction) were directly present in the ADR text and would have been surfaced by reading the cited ADRs end-to-end.
This is distinct from #638 (following ADR references discovered in code during investigation — here the ADRs are cited in documentation), #265 (detecting misleading narratives from individually-correct facts — here the issue is omitted provisions, not misleading framing), and #528 (verifying references are resolvable — here the links work, the content is incomplete).
Proposed change
In skills/pr-review/sub-agents/correctness.md in the fullsend-ai/agents repo, extend the 'Technical documentation with correctness surface area' section (or add a sibling 'Source material completeness' section) with guidance:
-
Completeness check for cited sources: When a documentation change cites authoritative source material by name or link (ADRs, RFCs, design docs, specs), read each cited source. After verifying the accuracy of stated claims, check whether the documentation omits any significant provisions, exceptions, or scoping constraints from those sources. An omitted exception (e.g., a carve-out for CI-runtime values) is as important as an inaccurate claim — it leaves reviewers with incomplete guidance that can produce the same false-positive outcomes the documentation exists to prevent.
-
Scope verification: When documentation establishes a rule (e.g., 'do not flag X'), check the cited source material for scoping or boundary conditions that the rule should acknowledge. If the source distinguishes between cases where the rule applies and cases where it does not, the documentation should reflect that distinction.
This guidance should apply regardless of whether the PR is docs-only or includes code changes. The key trigger is the presence of ADR/spec citations in the changed documentation.
Validation criteria
On the next 5 PRs to fullsend-ai/agents (or repos using this review agent) that add or modify documentation citing ADRs or design specs, the correctness sub-agent's transcript should show it reading the cited ADRs (observable as Read tool calls to ADR files) and checking for omitted provisions. Success: zero cases where a human reviewer subsequently identifies a missing ADR provision that the review agent did not flag, given that the provision was present in the cited ADR text.
Generated by retro agent from #731
What happened
On PR #731, the review agent ran 3 times (run 31424224434, run 31501630595, run 31638890823). The PR added AGENTS.md guidance summarizing ADR 0080 and ADR 0081 rules about harness env var literal defaults. The review agent's correctness sub-agent verified that stated claims were technically sound — confirming harness env var literal semantics,
base:composition overrides, and${VAR}expansion behavior against actual harness files. However, all 3 runs produced only a protected-path governance finding. Human reviewer waynesun09 then identified two substantive gaps by reading the cited ADRs: (1) the guidance omitted ADR 0081's explicit CI-runtime exception (values derived fromgithub.event.*, build matrix variables, or non-static secrets may live in the workflowenv:block), and (2) the rule was not scoped to distinguish static tuning defaults from per-run computed passthrough values likeCODE_ALLOWED_TARGET_BRANCHES. Both findings required comparing the documentation's summary against the full text of the referenced ADRs.What could go better
The correctness sub-agent's current guidance includes 'design document alignment — if the plan references a design document or ADR, are the claims consistent with the referenced source.' This check verifies accuracy (are stated claims correct?) but not completeness (does the summary cover all significant provisions?). On this PR, the stated claims were accurate — the review agent confirmed this. The gap was that the documentation omitted two significant ADR provisions entirely. The correctness sub-agent did not read ADR 0080 or ADR 0081 to check whether the AGENTS.md summary faithfully covered all their key rules, exceptions, and scoping.
Confidence is high: the review agent's own comment states 'the correctness sub-agent verified all claims,' confirming the accuracy check ran but completeness was not evaluated. Both human findings (ADR 0081 exception, scope distinction) were directly present in the ADR text and would have been surfaced by reading the cited ADRs end-to-end.
This is distinct from #638 (following ADR references discovered in code during investigation — here the ADRs are cited in documentation), #265 (detecting misleading narratives from individually-correct facts — here the issue is omitted provisions, not misleading framing), and #528 (verifying references are resolvable — here the links work, the content is incomplete).
Proposed change
In
skills/pr-review/sub-agents/correctness.mdin thefullsend-ai/agentsrepo, extend the 'Technical documentation with correctness surface area' section (or add a sibling 'Source material completeness' section) with guidance:Completeness check for cited sources: When a documentation change cites authoritative source material by name or link (ADRs, RFCs, design docs, specs), read each cited source. After verifying the accuracy of stated claims, check whether the documentation omits any significant provisions, exceptions, or scoping constraints from those sources. An omitted exception (e.g., a carve-out for CI-runtime values) is as important as an inaccurate claim — it leaves reviewers with incomplete guidance that can produce the same false-positive outcomes the documentation exists to prevent.
Scope verification: When documentation establishes a rule (e.g., 'do not flag X'), check the cited source material for scoping or boundary conditions that the rule should acknowledge. If the source distinguishes between cases where the rule applies and cases where it does not, the documentation should reflect that distinction.
This guidance should apply regardless of whether the PR is docs-only or includes code changes. The key trigger is the presence of ADR/spec citations in the changed documentation.
Validation criteria
On the next 5 PRs to fullsend-ai/agents (or repos using this review agent) that add or modify documentation citing ADRs or design specs, the correctness sub-agent's transcript should show it reading the cited ADRs (observable as Read tool calls to ADR files) and checking for omitted provisions. Success: zero cases where a human reviewer subsequently identifies a missing ADR provision that the review agent did not flag, given that the provision was present in the cited ADR text.
Generated by retro agent from #731