v1.7.1 optimizer hardening: stale re-scoring + assigned-challenger verdict + one-shot promote nudge#75
Merged
Merged
Conversation
…ibution + one-shot promote nudge) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…d_at field (v1.7.1)
…ution start (v1.7.1)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR hardens the v1.7.x optimizer/shadow-A/B loop by fixing three correctness gaps: stale-candidate re-scoring after champion swaps, verdict attribution to the assigned challenger (from shadow.json), and eliminating repeated PROMOTE nudges via a one-shot recommendation stamp. It’s an additive patch release (schema remains 1) with new regression tests across the prompt-pool, optimize-prompt, and conductor suites.
Changes:
- Add stale active candidate re-scoring at the start of
Invoke-PromptEvolution, and surfacerescoredresults in the evolution return contract and CLI output. - Add
Get-ShadowVerdict -ChallengerIdand plumb it throughComplete-Runso dollars verdicts evaluate the variant that actually ran. - Add
promote_recommended_atto pool records and stamp it once to ensure the PROMOTE nudge fires only once per candidate; update docs + plugin version.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| scripts/prompt-pool-lib.ps1 | Adds promote_recommended_at field handling and Get-ShadowVerdict -ChallengerId to pin attribution to the assigned challenger. |
| scripts/optimize-prompt-lib.ps1 | Implements stale-candidate re-scoring against the current champion and returns rescored in all result paths. |
| scripts/conductor-lib.ps1 | Pins shadow verdict evaluation to the assigned challenger and makes the PROMOTE event one-shot via promote_recommended_at. |
| scripts/fleet-optimize-prompt.ps1 | Prints rescored entries and shows the promotion recommendation timestamp in the pool verdict footer. |
| scripts/test-prompt-pool-lib.ps1 | Adds regression checks for promote_recommended_at round-trip and -ChallengerId behavior. |
| scripts/test-optimize-prompt-lib.ps1 | Adds regression checks ensuring stale candidates are re-scored and persisted (including failure-path persistence and unreadable-file skip). |
| scripts/test-conductor-lib.ps1 | Adds regression checks for one-shot PROMOTE nudge and assigned-challenger verdict attribution behavior. |
| docs/superpowers/specs/2026-07-03-optimizer-v171-hardening-design.md | New design spec describing the three fixes and their constraints. |
| docs/superpowers/plans/2026-07-03-optimizer-v171-hardening.md | New implementation plan capturing tasks, constraints, and test expectations. |
| docs/COMMANDS.md | Updates command docs to mention stale re-scoring and assigned-challenger attribution. |
| commands/optimize-prompt.md | Documents one-shot PROMOTE nudge behavior and stale re-scoring semantics/spend boundaries. |
| .claude-plugin/plugin.json | Bumps plugin version to 1.7.1-rc.1. |
Comment on lines
264
to
+268
| fail-opens to "no proposal this generation" with an honest reason; the | ||
| manifest is saved once per generation, never mid-flight. | ||
| .NOTES | ||
| Returns @{ success; applied; candidate_path; reason; generations }. | ||
| Returns @{ success; applied; candidate_path; reason; generations; rescored }. | ||
| rescored = @(@{ id; win_rate }, …) for any stale actives re-scored vs |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes v1.7.0's three named loose ends (spec: docs/superpowers/specs/2026-07-03-optimizer-v171-hardening-design.md, decision d073). Patch release; schema stays 1, all additions additive, zero migration.
The three fixes
--applychampion swap nulled other candidates' win rates, nothing ever re-scored them — one swap permanently darkened the pool. Now the next evolution run re-scores every stale active against the CURRENT champion via the existing minibatch evaluator, before evolving. Spend only in explicit/baton:optimize-promptruns, never on/baton:go. Newrescoredresult field surfaced by the runner.Complete-Runnow passes shadow.json'schallenger_idtoGet-ShadowVerdict -ChallengerId, so live dollars are judged against the variant pair that actually ran; assigned-but-retired →no-challenger, no action.promote_recommended_atstamp — the run-log nudge fires once per candidate;--poolstill shows the live verdict every time (now with the recommendation timestamp).Review
Final whole-branch review (opus): READY TO MERGE — 0 Critical / 0 Important / 2 Minor. M2 (unguarded champion read in the re-score pass) fixed pre-merge in 2c65eeb; M1 (release-notes file) is written at promotion by design. Full review:
.superpowers/sdd/final-review-v171.md.Tests
All five suites green after every task and after the M2 fix: prompt-pool 61 checks (P56–P61 new), optimize-prompt (E16/E17 new), conductor (SB11/SB12 new), bootstrap, baton-home. SB12 verified red-before/green-after against the misattribution scenario.
Plugin
1.7.0→1.7.1-rc.1.🤖 Generated with Claude Code