Effective cost — slice 2 (per-worker leaderboard) + v1.4 band fix#68
Merged
Conversation
Task 0: Get-QualityScalar lower bands half-open at report precision + in-band clamp, so a clean polish (0.6999) can never tie a pathological many-minor accept at 0.7 (d059 deferred item 1). Tests E2/E6b/E6c. Slice 2: Get-WorkerEffectiveCost folds per-run effective-cost.json records into a per-worker learned leaderboard (whole-run, single-producer-weighted, confidence-gated) + Format-EffectiveCostLeaderboard (E34-E46); new /baton:effective-cost surface (fleet-effective-cost.ps1 + commands/effective-cost.md, hermetic CLI test C1-C10); bootstrap manifest + assert; plugin 1.4.0-rc.3 -> rc.4. Advisory/legibility only — no routing change. Box-private records stay under $BATON_HOME/runs. The confidence-gated Select-Capability re-rank remains a named, deferred future slice. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e Stitch Kevin's design exploration for the Baton dashboard: Live Fleet Ops (preferred nav), System Intelligence, and Cost Ledger — each with code.html, screen.png, and a DESIGN.md design system. Parked for integration when ready. The System Intelligence MODEL LEADERBOARD panel is the future visual home for the effective-cost slice-2 leaderboard. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ConvertTo-Json on a piped array unrolls it, so the single-row board serialized as a bare object and the empty board as "" — breaking the documented rows-array contract the dashboard consumes. Use -InputObject @($board). Tests C11-C13 pin 1-row -> array and 0-row -> []. (Final-review Important.) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Implements “effective cost” slice 2 by folding per-run effective-cost.json artifacts into a per-worker leaderboard, adds a quality-band boundary fix to ensure strict cross-band ordering at report precision, and exposes the leaderboard via a new /baton:effective-cost command (CLI + docs + tests). This PR also introduces a set of dashboard design prototype assets under dashboard/Design from Google stitch/..., which is not currently described in the PR description.
Changes:
- Adjust
Get-QualityScalarband boundaries to be half-open (at 4dp) for non-accept bands and clamp in-band. - Add
Get-WorkerEffectiveCost(fold) +Format-EffectiveCostLeaderboard(renderer), plus a newfleet-effective-cost.ps1CLI and hermetic tests. - Update bootstrap deployment manifest/tests and bump plugin version to
1.4.0-rc.4.
Reviewed changes
Copilot reviewed 17 out of 23 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/test-fleet-effective-cost.ps1 | New hermetic CLI test for fleet-effective-cost.ps1 including --json array contract and malformed-record skip behavior. |
| scripts/test-effective-cost-lib.ps1 | Updates quality-band expectations and adds tests for the new per-worker fold + leaderboard formatter. |
| scripts/test-bootstrap.ps1 | Ensures bootstrap dry-run output includes fleet-effective-cost.ps1. |
| scripts/fleet-effective-cost.ps1 | New CLI surface that reads per-run artifacts, folds via Get-WorkerEffectiveCost, and renders/prints JSON rows. |
| scripts/effective-cost-lib.ps1 | Band-boundary fix in Get-QualityScalar; adds Get-WorkerEffectiveCost and Format-EffectiveCostLeaderboard. |
| scripts/bootstrap.ps1 | Adds fleet-effective-cost.ps1 to deployed script manifest. |
| docs/superpowers/plans/2026-06-25-effective-cost-metric-slice2.md | Implementation plan for slice 2 (leaderboard + CLI + tests + bootstrap bump). |
| commands/effective-cost.md | New /baton:effective-cost command documentation and usage patterns. |
| .claude-plugin/plugin.json | Bumps plugin version 1.4.0-rc.3 → 1.4.0-rc.4. |
| dashboard/Design from Google stitch/DESIGN.md | Adds dashboard design system/prototype documentation (new). |
| dashboard/Design from Google stitch/Some notes.txt | Adds notes about prototype layouts and integration (new). |
| dashboard/Design from Google stitch/Live Fleet Ops/DESIGN.md | Adds “Live Fleet Ops” prototype design doc (new). |
| dashboard/Design from Google stitch/Live Fleet Ops/code.html | Adds “Live Fleet Ops” prototype HTML (new). |
| dashboard/Design from Google stitch/Token Cost and Analytics/DESIGN.md | Adds “Token Cost and Analytics” prototype design doc (new). |
| dashboard/Design from Google stitch/Token Cost and Analytics/code.html | Adds “Token Cost and Analytics” prototype HTML (new). |
| dashboard/Design from Google stitch/Analytics and Insights/DESIGN.md | Adds “Analytics and Insights” prototype design doc (new). |
| dashboard/Design from Google stitch/Analytics and Insights/code.html | Adds “Analytics and Insights” prototype HTML (new). |
Comment on lines
+26
to
+33
| $pattern = if ($Glob) { $Glob } else { Join-Path $Root '*/effective-cost.json' } | ||
| $records = @() | ||
| foreach ($f in @(Get-ChildItem -Path $pattern -File -ErrorAction SilentlyContinue)) { | ||
| try { $records += (Get-Content -LiteralPath $f.FullName -Raw | ConvertFrom-Json) } | ||
| catch { [Console]::Error.WriteLine("skipped unreadable record: $($f.FullName)") } | ||
| } | ||
| return ,@($records) | ||
| } |
Comment on lines
+8
to
+12
| The learned cost-vs-quality scoreboard. Each `/baton:go` run that passes through | ||
| the acceptance gate writes an `effective-cost.json` record (`effective_cost = | ||
| realized_cost ÷ realized_quality` — what you actually paid for the quality you | ||
| got, lower is better). This command folds those per-run records into a | ||
| **per-worker leaderboard**, ranked cheapest-quality-adjusted first, so you can |
Comment on lines
+1
to
+5
| These are three possible layouts for the Baton Dashboard. | ||
|
|
||
| The Cockpit - live fleet ops. | ||
| The cockpit - Analytics and Insights (Intelligence) | ||
| The Cockpit - Token & Cost Analytics. I prefer the Live Fleet Ops dashboard menu over the other two. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Effective cost — slice 2
Folds the per-run
effective-cost.jsonrecords (slice 1's join artifact) into a per-worker learned leaderboard and exposes it as/baton:effective-cost. Advisory / legibility only — no routing change.What's in it
Get-QualityScalarlower bands are now half-open at report precision + in-band clamp, so a clean polish (0.6999) can never tie a pathological many-minor accept at 0.7. Exhaustive sweep in review confirmed strict cross-band ordering for all count combos.Get-WorkerEffectiveCost— pure fold, whole-run single-producer-weighted, confidence-gated.Format-EffectiveCostLeaderboard— pure renderer (cheapest-first, tentative flag for low confidence)./baton:effective-cost—fleet-effective-cost.ps1+commands/effective-cost.md.--jsonemits the rows array the future dashboard MODEL LEADERBOARD panel consumes.1.4.0-rc.3 → rc.4.Tests
test-effective-cost-lib.ps1E1–E46,test-fleet-effective-cost.ps1C1–C13 (incl. the 0/1-row--jsonarray contract),test-conductor-lib.ps1T1–T88,test-bootstrap.ps1— all green.Review
Final adversarial opus review: 0 Critical, 1 Important (the
--jsonsingle/empty-row array trap) fixed (cc5a75f,ConvertTo-Json -InputObject), 0 Minor blocking. Band fix + fold math + box-private/advisory invariants verified clean.Box-private: records stay under
$BATON_HOME/runs;references/fleet.yamluntouched. The confidence-gatedSelect-Capabilityre-rank remains a named, deferred future slice.🤖 Generated with Claude Code