Skip to content

docs: record Pi pilot and separate agent evaluation from regression - #25

Merged
aDragon0707 merged 1 commit into
mainfrom
codex/adr-pi-evaluation
Oct 15, 2026
Merged

aDragon0707 merged 1 commit into
mainfrom
codex/adr-pi-evaluation

Conversation

@aDragon0707

@aDragon0707 aDragon0707 commented Oct 15, 2026 •

Copy link
Copy Markdown
Owner

Summary

The existing harness constructs fixtures for SACP's engine; it does not run an agent. That distinction was too easy to miss. Record the accepted decision and first pilot choice:

Keep deterministic engine scenarios in ordinary CI
Add a separate real-agent evaluation layer
  Pi JSON subprocess → independent collection → task acceptance + SACP + baseline

Adds ADR-0001 and a focused glossary; aligns both READMEs, both roadmaps, status, architecture and development docs. Pi is selected for the first bounded integration pilot, not claimed as implemented or benchmarked. Provider/model, execution budget, installed runtime pin and isolated environment remain unresolved.

Evidence

  • Before: source directly creates claims/mock events, but the top-level “offline harness” framing obscured its test subject.
  • After: current scenario regression, Pi's agent harness and the planned evaluation harness are explicitly distinguished. The ADR defines independent acceptance, comparison denominators, injected-fault separation, stop rules and unchanged deterministic CI contracts.
  • Official Pi source checked at b06c1f2cf1dea6af8748ddeee18c23587d2cb7c6: README/package.json, CLI, JSON events, RPC, security and environment docs. The old badlogic/pi-mono URL resolves to earendil-works/pi. JSON/RPC support is documented; no installation or model call was performed.
  • Local documentation links/fences/public-path checks and git diff --check pass. Existing python -m sacp_harness is rechecked without adding tests or dependencies for this docs-only decision.
  • Pre-merge review: base 860444e, head a49f238. All eight CI jobs 38684250650 pass; receipt 38684293710 is COMPLETE/200, strict PASS, matching current head, without policy warnings/evidence errors. Standards: vocabulary, relative links and paired README/roadmap updates are consistent. Spec: preserves deterministic regression, accepts a separate Pi pilot, records alternatives, independent oracle, metrics, lifecycle/security limits and implementation gates. No blocking findings; manually squash merge at this verified head, then validate main.
  • Post-merge: main c06106b, verification 38684540632, all seven jobs success. Downloaded all six suite artifacts and receipt; checked exact inventories (core 100 per environment, LangGraph 7, protocol 49, harness 8 plus 3 controls), zero failures/errors/skips, SHA/run binding and independent Git tree fingerprint. Receipt COMPLETE/200, strict PASS. No Pi/model execution occurred; branch retained.

Merge Danger

Door: two-way

Blast Radius: documentation

No agent runner, provider credential, package install, model call, new required job or core protocol field is introduced. Accepting the decision does not authorize paid runs or public session uploads. Pi's interface/identity can evolve; fixed source citations and a required feasibility spike bound the choice.

@aDragon0707
aDragon0707 merged commit c06106b into main Oct 15, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant