Fleet does the labor — Slice 1: round-trip proof + prompt hardening (v1.10.0-rc.1)#79
Merged
Merged
Conversation
…t-does-the-labor track) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…hardening Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… empty-prompt binding
Task 3 routed stdin:true providers through Resolve-FleetCommand -Prompt '',
which a mandatory [string] param rejects, breaking every stdin:true provider
(github-models, box-private reviewer pairs). Resolve {{model}} inline for both
stdin paths instead. Add a stdin:true fixture provider + round-trip asserts —
the gap that let this through review.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…G token Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Implements “Fleet does the labor” Slice 1 by adding a live round-trip probe (fleet doctor --live) that verifies enabled providers actually answer a deterministic canary battery, and hardens CLI dispatch by promoting certain templates to stdin-based prompting to avoid quoting/965-byte issues.
Changes:
- Add
scripts/fleet-probe-lib.ps1with canary battery scoring, reachability checks, and timeout-guarded live probing. - Extend
fleet doctorwith-Live/-TimeoutS/-Jsonto reportlive_ok | live_fail(reason) | skipand exit nonzero if any enabled provider fails. - Promote “clean-tail quoted
{{prompt}}” CLI templates to stdin dispatch viaTest-StdinSafe, plus fixtures/tests/docs/version bump and deploy wiring.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/fleet-probe-lib.ps1 | New canary battery + reachability + timeout-guarded probe implementation. |
| scripts/test-fleet-probe-lib.ps1 | Unit tests for canary scoring, reachability, and probe behavior. |
| scripts/fleet-lib.ps1 | Adds Test-StdinSafe and promotes eligible CLI templates to stdin dispatch. |
| scripts/test-fleet-dispatch.ps1 | Tests stdin-safe predicate and stdin:true regression coverage. |
| scripts/fleet-doctor.ps1 | Adds -Live probing mode and JSON/table rendering for live results. |
| scripts/test-fleet-doctor.ps1 | End-to-end tests for fleet doctor --live, including exit codes and JSON shape. |
| scripts/fixtures/fleet-sample.yaml | Adds stub-stdin provider fixture for stdin:true coverage. |
| scripts/test-fleet-lib.ps1 | Updates provider-count assertion for the expanded fixture roster. |
| scripts/bootstrap.ps1 | Deploy manifest updated to include fleet-probe-lib.ps1. |
| scripts/test-bootstrap.ps1 | Adds deploy assertion for fleet-probe-lib.ps1. |
| docs/superpowers/specs/2026-07-04-fleet-labor-slice1-roundtrip-proof-design.md | New design spec documenting live probing + prompt hardening approach. |
| docs/superpowers/plans/2026-07-04-fleet-labor-slice1-roundtrip-proof.md | New implementation plan (task breakdown, tests, wiring, docs). |
| commands/fleet.md | Updates /baton:fleet docs for doctor --live usage and hints. |
| AGENTS.md | Adds a fleet health note recommending fleet doctor --live. |
| .claude-plugin/plugin.json | Version bump to 1.10.0-rc.1. |
Comment on lines
+46
to
+52
| $up = ([string]$Output).ToUpperInvariant() | ||
| $score = 0 | ||
| $missing = @() | ||
| foreach ($ch in $script:FleetCanaryChallenges) { | ||
| if ($up.Contains(([string]$ch.token).ToUpperInvariant())) { $score++ } | ||
| else { $missing += [string]$ch.token } | ||
| } |
Comment on lines
+72
to
+76
| if ($Provider.kind -eq 'cli') { | ||
| $bin = ([string]$Provider.command_template -split '\s+')[0] | ||
| if (Get-Command $bin -ErrorAction SilentlyContinue) { return @{ reachable = $true; reason = $null } } | ||
| return @{ reachable = $false; reason = 'not-on-PATH' } | ||
| } |
Comment on lines
+193
to
+195
| if ($template -notmatch '\s+(["''])\{\{prompt\}\}\1\s*$') { return $false } | ||
| $head = $template -replace '\s+(["''])\{\{prompt\}\}\1\s*$', '' | ||
| if ($head -match '[|><&;`]' -or $head -match '\$\(') { return $false } |
| --- | ||
| description: Manage and invoke the LLM fleet. `doctor` health-checks providers, `test` dispatches a prompt to one provider, `list` shows the registry. | ||
| argument-hint: doctor | test <name> "<prompt>" [--model <m>] | list | ||
| argument-hint: doctor [--live] [--timeout <s>] | test <name> "<prompt>" [--model <m>] | list |
Comment on lines
+21
to
+24
| & pwsh -NoProfile -File "$HOME/.claude/scripts/fleet-doctor.ps1" -Live -TimeoutS 60 | ||
| ``` | ||
|
|
||
| Echo the table to the user. | ||
| Echo the table to the user. With `--live`, each enabled provider receives a `PONG` canary and reports `live_ok`, `live_fail(<reason>)`, or `skip`; a `no-canary` reason means the provider ran but didn't answer (often a wrong command template for this box). Plain (no `--live`) performs the fast PATH/reachability check only. |
| - Guard `0/0` NaN in any division (this slice has none — assert none is introduced). | ||
| - Box-private: never write real roster/endpoint values into the shared seed `fleet.yaml`; placeholder hosts only. The live probe reads the box-private live roster at run time. | ||
| - Tests are hermetic: temp `fleet.yaml` fixtures, temp `BATON_HOME`, `try/finally` restore. NEVER touch real `~/.baton`, `~/.claude`, `D:\Dev\Grimdex`, or `D:\dev`. | ||
| - The canary prompt constant is exactly `Reply with exactly the word PONG and nothing else.` and the token constant is exactly `PONG`. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fleet Does the Labor — Slice 1: Round-Trip Proof + Prompt Hardening (v1.10.0-rc.1)
First slice of the "fleet does the labor" track: prove that each enabled instrument actually answers a real prompt, and harden the dispatch pipe — so a later slice can build the executor on a pipe known to work. No repo-apply here — the
-Spawnerexecutor is Slice 2.Spec:
docs/superpowers/specs/2026-07-04-fleet-labor-slice1-roundtrip-proof-design.md(d077). Plan:docs/superpowers/plans/2026-07-04-fleet-labor-slice1-roundtrip-proof.md.What's in it
fleet doctor --live— sends a 4-challenge canary battery to every ENABLED provider (cli + http) in one round-trip and reportslive_ok | live_fail(reason) | skip. The battery (PONG/42/PARIS/COLD) is scored k/4; onlyPONG's answer appears in the prompt, so an echoing/wrong template scores 1/4 and correctly fails.--timeout <s>(default 60) and--jsonsupported. Exit 0 iff every enabled provider islive_ok, else 1. Default (non-live) path unchanged."{{prompt}}") now dispatch via stdin, immune to the 965-byte / quote-mangling wall. Embedded-{{prompt}}and shell-wrapped templates stay on the legacy path.scripts/fleet-probe-lib.ps1(canary classifier + reachability + timeout-guarded live probe), deploy-wired intobootstrap.ps1with atest-bootstrapdeploy assert. Docs:commands/fleet.md,AGENTS.md. Plugin1.9.0 → 1.10.0-rc.1.Quality
Two fixes after the review
c6b9764) — the review found the singlePONGtoken lived inside its own prompt, so an echo-back false-passedlive_ok. Replaced with the 4-challenge battery; echo-back now scores 1/4 and fails.5bec3e6) — a Critical the review missed: Task 3 routedstdin:trueproviders throughResolve-FleetCommand -Prompt '', which a mandatory[string]param rejects, breaking every stdin:true provider (github-models, box-private reviewer pairs). The fixture had no stdin:true provider, so the suites didn't catch it. Fixed (resolve{{model}}inline, no empty-prompt binding) + added astub-stdinfixture and round-trip asserts to close the coverage gap.Upgrade note
The stdin promotion changes dispatch for real interpolating providers whose template ends in a quoted
{{prompt}}(codex exec,gh copilot suggest,ollama run,agy --print,claude -p): they now receive the prompt on stdin. Re-runfleet doctor --liveafter upgrading to confirm your box's roster still answers.Out of scope (Slice 2 / later)
The
-Spawnerexecutor that applies repo changes; routing changes; per-provider latency/quality benchmarking.