From a49f23801c7d744c580729eb5235d611d9255999 Mon Sep 17 00:00:00 2001 From: aDragon0707 Date: Thu, 15 Oct 2026 12:58:00 +0800 Subject: [PATCH] docs: record Pi pilot and separate agent evaluation from regression --- GLOSSARY.md | 26 +++++ README.md | 7 +- README.zh-CN.md | 7 +- ROADMAP.md | 6 + ROADMAP.zh-CN.md | 6 + STATUS.md | 4 +- .../adr/0001-real-agent-evaluation-with-pi.md | 104 ++++++++++++++++++ docs/architecture.md | 6 +- docs/development.md | 6 + 9 files changed, 165 insertions(+), 7 deletions(-) create mode 100644 GLOSSARY.md create mode 100644 docs/adr/0001-real-agent-evaluation-with-pi.md diff --git a/GLOSSARY.md b/GLOSSARY.md new file mode 100644 index 0000000..2c9547b --- /dev/null +++ b/GLOSSARY.md @@ -0,0 +1,26 @@ +# SACP evaluation vocabulary + +本页区分 SACP 的验证对象与评测角色。实现选择与执行边界见 [ADR-0001](./docs/adr/0001-real-agent-evaluation-with-pi.md)。 + +## Language + +**场景回归测试（Scenario regression harness）**：以明确构造的声明和证据为输入，检查 SACP 判定是否符合预期的测试集合。 +_Avoid_: 真实 Agent 评测、Agent 能力 benchmark + +**Agent 运行框架（Agent harness）**：承载模型调用、工具执行和会话生命周期的运行系统；在评测中是被驱动和观察的对象。 +_Avoid_: 评测器、独立验收器 + +**真实 Agent 评测（Agent evaluation harness）**：围绕真实 Agent 执行任务，采集记录、组织独立验收并比较判定结果的评测系统。 +_Avoid_: 仅凭模拟事件得到的真实准确率 + +**独立任务验收（Independent task acceptance）**：依据预先确定的任务条件，对实际交付进行的检查；不以 Agent 自述或 SACP 的判定作为正确性标准。 +_Avoid_: Agent 自证、以被测规则作为标准答案 + +**任务完成（Task completion）**：实际交付满足该任务的验收条件。 +_Avoid_: 进程正常退出、会话结束、测试命令返回成功 + +**完成声明（Completion claim）**：Agent 对任务或步骤已经完成的陈述，其支持程度须与对应证据分别判断。 +_Avoid_: 完成证明 + +**预检通过（Preflight pass）**：记录输入满足指定证据与授权契约，允许调用方进入下一步。 +_Avoid_: 任务必然正确、外部动作已成功、自动获准发布 diff --git a/README.md b/README.md index 8b41afa..132c531 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,9 @@ Positive controls passed: 3/3 ## Protocol validator: 49/49 strict PASS (0 FAIL) ``` -The harness uses fixed synthetic scenarios without model calls. It prints by default; `--output-dir .local/harness` explicitly writes reports. Continue through the [example guide](./examples/README.md) for deployment, refund, recovery and optional LangGraph demos. +This is a **scenario regression harness for the SACP engine**: it constructs claims and mock events directly, without running a coding agent or calling a model. It prints by default; `--output-dir .local/harness` explicitly writes reports. Continue through the [example guide](./examples/README.md) for deployment, refund, recovery and optional LangGraph demos. + +A separate real-agent evaluation layer is planned, with Pi as the first integration pilot. It is not implemented or benchmarked yet; see [ADR-0001](./docs/adr/0001-real-agent-evaluation-with-pi.md) for the decision, independent acceptance criteria and limits. @@ -89,7 +91,7 @@ flowchart TB Verdict --> Gate[Release / deployment / refund gate] Verify --> Recovery[Reconciliation route and next owner] end - subgraph Offline[Offline harness: verify explicit boundaries] + subgraph Offline[Scenario regression: verify engine boundaries] Cases[Breakers and positive controls] end Cases -. Exercise .-> Online @@ -147,6 +149,7 @@ The base engine uses Python's standard library; protocol/CI tooling requires PyY ## 📚 Documentation and contributing - **Understand:** [Architecture](./docs/architecture.md) · [Evidence model](./docs/evidence-model.md) · [State projection](./docs/state-machine.md) +- **Evaluation decisions:** [Vocabulary](./GLOSSARY.md) · [ADR-0001: real-agent evaluation with Pi](./docs/adr/0001-real-agent-evaluation-with-pi.md) - **Run and develop:** [Examples](./examples/README.md) · [Development and suite commands](./docs/development.md) · [Runtime contracts](./docs/runtime-contracts.md) - **Review and extend:** [Maintainer workflow](./docs/maintainer-workflow.md) · [Dirty Run cases](./DIRTY_RUN_CASES.md) · [Contributing](./CONTRIBUTING.md) diff --git a/README.zh-CN.md b/README.zh-CN.md index 9efecc3..adc96fe 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -70,7 +70,9 @@ Positive controls passed: 3/3 ## Protocol validator: 49/49 strict PASS (0 FAIL) ``` -Harness 使用固定合成场景，不调用模型。默认只打印，显式传入 `--output-dir .local/harness` 才写报告。接着按[示例导航](./examples/README.zh-CN.md)体验部署、退款、恢复和可选 LangGraph demo。 +这里是 **SACP 引擎的场景回归测试（Harness）**：直接构造声明与模拟事件，不运行 coding agent，也不调用模型。默认只打印，显式传入 `--output-dir .local/harness` 才写报告。接着按[示例导航](./examples/README.zh-CN.md)体验部署、退款、恢复和可选 LangGraph demo。 + +另计划新增真实 Agent 评测层，以 Pi 为首个接入试点。目前尚未实现，也没有对应 benchmark 结果；选择理由、独立验收标准与边界见 [ADR-0001](./docs/adr/0001-real-agent-evaluation-with-pi.md)。 @@ -89,7 +91,7 @@ flowchart TB Verdict --> Gate[发布 / 部署 / 退款门禁] Verify --> Recovery[对账路线与下一责任人] end - subgraph Offline[离线 Harness：验证明确的边界] + subgraph Offline[场景回归测试：验证引擎边界] Cases[反例与正向控制] end Cases -. 测试 .-> Online @@ -147,6 +149,7 @@ PR 代码 → 只读 CI → 证据 artifact ## 📚 文档与贡献 - **理解项目：**[架构](./docs/architecture.md) · [证据模型](./docs/evidence-model.md) · [状态投影](./docs/state-machine.md) +- **评测决策：**[术语](./GLOSSARY.md) · [ADR-0001：真实 Agent 评测与 Pi 选择](./docs/adr/0001-real-agent-evaluation-with-pi.md) - **运行与开发：**[示例](./examples/README.zh-CN.md) · [开发与 suite 命令](./docs/development.md) · [运行时契约](./docs/runtime-contracts.md) - **审查与扩展：**[维护者流程](./docs/maintainer-workflow.md) · [Dirty Run cases](./DIRTY_RUN_CASES.md) · [贡献指南](./CONTRIBUTING.zh-CN.md) diff --git a/ROADMAP.md b/ROADMAP.md index 4c01819..5ac421e 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -10,6 +10,12 @@ The text-first spec, validator, dirty cases, Receipt Chain profile, local Doctor CI runs a cross-platform core matrix, explicit LangGraph integration, strict protocol validation and a deterministic harness with positive controls. Trusted default-branch code cross-checks GitHub execution facts and current PR identity before constructing a receipt. Merging remains a manual maintainer action. See [architecture](./docs/architecture.md) and [verification scope](./docs/verification-2026-10-15.md). +## Next: real-agent evaluation pilot + +[ADR-0001](./docs/adr/0001-real-agent-evaluation-with-pi.md) retains the existing deterministic engine scenarios and selects Pi as the first real-agent pilot. Implement a Python outer runner using Pi's JSON subprocess interface, isolate and pin its environment, then collect actual coding-task trajectories and independently verify the delivered patches. The adapter, provider/model, budget and real results are not yet available. + +Start with a small fixed task set and compare SACP with a predefined test-based baseline on the same records. Report missed false completions, incorrect blocks, indeterminate outcomes and overhead; separate injected faults from naturally occurring failures. Keep model calls out of required PR CI. Acceptance stages and stop conditions are defined in the ADR. + ## Next: trust and independent acceptance - Complete an actual controlled fork PR smoke run; synthetic fork API tests are not equivalent. diff --git a/ROADMAP.zh-CN.md b/ROADMAP.zh-CN.md index 1d3e83f..3f085bb 100644 --- a/ROADMAP.zh-CN.md +++ b/ROADMAP.zh-CN.md @@ -10,6 +10,12 @@ SACP/0.1 是当前交换契约。下面描述能力成熟度，不意味着每 CI 已运行跨平台 core 矩阵、显式安装依赖的 LangGraph 集成、严格协议校验，以及包含正向控制的确定性 harness。默认分支上的可信实现交叉核对 GitHub 执行事实和当前 PR 身份，再生成 receipt。合并仍由维护者手动发起。参见[架构](./docs/architecture.md)与[验证范围](./docs/verification-2026-10-15.md)。 +## 下一步：真实 Agent 评测试点 + +[ADR-0001](./docs/adr/0001-real-agent-evaluation-with-pi.md)决定保留现有确定性引擎场景测试，以 Pi 为首个真实 Agent 试点。计划使用 Python 外层 runner 与 Pi JSON 子进程接口，固定并隔离环境，采集实际代码任务轨迹并独立验收交付补丁。目前尚无 adapter、选定的 provider/model、运行预算或真实评测结果。 + +先用小规模固定任务集，在同一批记录上比较 SACP 与预先定义的测试基线，报告漏检、误拦、不可判定和额外开销；注入故障与自然失败分别统计。模型调用不进入 required PR CI。分阶段验收与停止条件见 ADR。 + ## 下一步：可信度与独立验收 - 完成真实、受控的 fork PR 烟测；合成 fork API 测试不能代替它。 diff --git a/STATUS.md b/STATUS.md index 9c7a635..2d7f921 100644 --- a/STATUS.md +++ b/STATUS.md @@ -15,7 +15,7 @@ Start with [README](./README.md) / [中文](./README.zh-CN.md), then [architectu | Mock idempotency | Same-database refund reservations, conflict rejection, restart and concurrent calls | Simulated effect identity, not end-to-end exactly-once | | Adapters | REST-shaped GitHub payload mapping, trusted reviewer allowlist, CLI, real LangGraph wiring/checkpointers | Payload source authentication remains an integration responsibility | | HTTP experiments | Real localhost provider transport, simulated health/rollback | No public SACP API or production provider | -| Harness | Eight synthetic breakers and three positive controls | Fixed small set; no measured population accuracy | +| Scenario regression harness | Eight synthetic breakers and three positive controls for the SACP engine | Does not run a real agent/model; no measured population accuracy | | Governance | Read-only PR CI, trusted independent attestation, formal receipt, manual squash merge, main revalidation | Semantic review required; real fork E2E still outstanding | | Corpus and Doctor | 34 translated receipts, YAML examples, local text-audit CLI | Historical/synthetic inputs; Doctor uses heuristics and does not rerun tasks | @@ -33,4 +33,6 @@ The protocol itself does not require a database or HTTP transport. The Python re ## Next work +The accepted [real-agent evaluation decision](./docs/adr/0001-real-agent-evaluation-with-pi.md) preserves deterministic scenarios and selects Pi as the first coding-agent pilot, using an outer runner and independent task acceptance. This is planned work: no Pi adapter, selected provider/model or real-agent benchmark result exists yet. + Finish controlled real fork end-to-end acceptance, strengthen isolation against arbitrary malicious Python tests, authenticate external collectors where used, and gather independently reviewed real-world false-completion and false-block cases. Keep schema/transport expansion behind concrete dirty cases and compatibility review. See [ROADMAP](./ROADMAP.md). diff --git a/docs/adr/0001-real-agent-evaluation-with-pi.md b/docs/adr/0001-real-agent-evaluation-with-pi.md new file mode 100644 index 0000000..77e18fe --- /dev/null +++ b/docs/adr/0001-real-agent-evaluation-with-pi.md @@ -0,0 +1,104 @@ +--- +status: accepted +date: 2026-10-15 +--- + +# ADR-0001：保留场景回归测试，以 Pi 试点真实 Agent 评测 + +当前 `sacp_harness/` 直接构造 Claim、mock provider 事件和授权，验证 SACP 引擎及协议文档；没有驱动真实 Agent 或调用模型。我们决定保留它作为确定性场景回归测试，另增真实 Agent 端到端评测层，并选择 Pi coding agent 作为首个有界接入试点，以回答 SACP 相比已有测试检查增加了什么判别价值。 + +本 ADR 接受的是架构方向和首个试点对象。Pi adapter、执行环境、任务集和评测结果均尚未实现；源码可接入性已核对，安装兼容性、隔离和实际采集须经后续 spike 验收。 + +## 决策与术语 + +1. **保留现有场景回归测试。** `python -m sacp_harness` 的接口、8 个固定反例、3 个正向控制及现有 CI 契约继续保留，不改名目录，也不把模型调用引入 required PR jobs。 +2. **新增独立的真实 Agent 评测层。** 先覆盖代码修复的小任务；实际执行与已保存轨迹的离线回放分开。真实调用是显式启动、有预算的实验，不能通过普通 CI 或导入模块自动发生。 +3. **Pi 是首个 Agent 运行框架，不是模型或验收器。** 必须另外记录具体 provider、model、推理设置与工具配置；结果归属于该组合及任务集，不泛化为所有 Agent。 +4. **Python 外层以 Pi JSON 模式子进程为首版接缝。** 一次任务一次进程，采集 JSONL、stderr、退出原因与产物。暂不引入 TypeScript SDK；需要交互控制时再评估 RPC。 +5. **SACP 是被评估的判定层。** 独立任务验收器提供对照依据，不能用 SACP 自己的投影结果生成标准答案。Pi 的工具结果也只是待核对的执行记录。 + +术语定义见 [GLOSSARY](../../GLOSSARY.md)。Pi 官方使用的 “agent harness” 与本项目拟建的 “evaluation harness” 是不同角色。 + +## 为什么先选 Pi + +2026-10-15，GitHub API 将旧 `badlogic/pi-mono` 地址重定向到 [`earendil-works/pi`](https://github.com/earendil-works/pi)。本次检查固定于源码 commit [`b06c1f2`](https://github.com/earendil-works/pi/tree/b06c1f2cf1dea6af8748ddeee18c23587d2cb7c6)，不把浮动 main 当作长期接口保证。 + +| 已核对能力 | 对本项目的意义 | +|---|---| +| MIT 许可、可读源码 | 能审查事件和工具语义，保留版本化接入依据 | +| JSON 模式执行一次任务并输出 JSONL | 适合当前 Python 外层采集，首版不需要交互式终端自动化 | +| RPC 与 TypeScript SDK 可选 | 后续若需要控制或嵌入，仍有明确接缝 | +| message、tool execution、retry、settled 等事件 | 能记录真实调用过程及失败阶段，而非只保存最终回复 | +| 可配置模型、工具、会话与扩展 | 有条件固定实验配置，减少个人环境污染 | + +检查的 `package.json` 标识包名 `@earendil-works/pi-coding-agent`、源码版本 `1.2.0`、Node.js `>=22.19.0`。这不是本项目已安装该版本或已验证 npm 发布产物的声明。实际试点必须锁定安装版本、依赖锁/完整性与事件契约。 + +选择 Pi 的理由是可观察和易接入，不是已有证据表明它的任务成功率优于其他 Agent。 + +## 边界与执行流程 + +```mermaid +flowchart LR + Task[固定任务与初始仓库 SHA] --> Runner[外层 runner] + Runner --> Pi[隔离环境内的 Pi 与固定模型] + Pi --> Trace[原始事件与最终产物快照] + Trace --> Evidence[宿主独立核对与证据归一化] + Evidence --> SACP[SACP 规则判定] + Trace --> Oracle[独立任务验收] + Trace --> Baseline[预先定义的测试基线] + Oracle --> Compare[比较漏检 / 拒绝 / 成本] + SACP --> Compare + Baseline --> Compare +``` + +- **固定输入：**任务说明、初始 SHA、可改路径、预置测试与隐藏验收条件先确定。Agent 不可修改外层 runner、验收器和证据存储；新增测试可作为交付，但不可替换固定验收。 +- **固定配置：**记录 Pi 和 Node 版本、provider/model ID、推理与生成设置、提示词/资源摘要、工具清单、依赖和执行环境。无法固定的随机性与 provider 版本漂移如实记录，不承诺逐字重现。 +- **隔离外部影响：**将 Pi 及其工具放入明确的 OS/container/VM 边界；普通 worktree、子进程、工作目录和工具 allowlist 本身不构成沙箱。仅暴露任务所需文件和模型访问；禁止继承个人插件、MCP、配置、GitHub token 或其他无关凭据。模型凭据通过受控边界提供，不进入任务文件、命令参数或公开日志。 +- **采集生命周期：**按 LF 解析 JSONL，以 toolCallId 关联工具事件，分别保存 stderr 与进程退出原因。不能把 RPC prompt 受理、`agent_end`、`agent_settled` 或 exit 0 单独当作任务完成；显式处理中止、provider error、超时、输出截断和进程异常。JSON 模式首版须核对 settled、流完整性与进程结束，之后仍执行独立验收。 +- **独立取证：**在运行结束后的固定快照上，由外层计算 patch/文件摘要，并从隔离的干净初始环境重建依赖、应用补丁、执行固定验收。记录测试身份、选择范围、collection/setup/断言阶段、exit code、skip 与产物；不能只相信 Agent 声称运行过测试或复用其可污染环境。 +- **分离结果：**分别记录任务成功/失败/不可判定、完成声明的支持情况、SACP 预检结果和运行基础设施状态。自然语言声明的提取与歧义处置要版本化，并保留人工复核；不能把所有失败归因给 Agent，也不能默认任何最终回复都声称全部完成。 +- **协议兼容：**优先映射现有 `tests_passed`、`artifact_built` 等已支持策略；未知 claim 保持 unverified。先用有类型的实验记录保存评测结果；若输出 SACP/0.1 receipt，须 strict 校验，实验元数据放 extensions，不凭本 ADR 新增核心字段或弱化政策。 + +## 最小试点与验收 + +首轮建议 10–20 个固定小任务，只作为探索试点。先锁定一个 Pi/模型组合，不先建设多 Agent 平台；具体 provider/model、真实运行预算与执行环境尚未选择。本 ADR 不授权付费调用、上传会话、自动发布或实际执行业务副作用。 + +| 阶段 | 必须交付的证据 | 停止条件 | +|---|---|---| +| 接入 spike | 固定运行时与事件样例；正常结束、工具错误、provider 错误、超时/中止的采集；隔离与凭据边界验证 | 生命周期不明、无法限制环境或无法可靠保存事件时，不开始批量真实运行 | +| 任务/验收准备 | 至少一个已知有效补丁和一个失败交付，证明独立验收可区分两者；任务与测试预先冻结 | 验收器无法区分或证据不足时，标不可判定并修复验收，不调 SACP 结果凑通过 | +| 真实试点 | 每次实际运行的配置、原始轨迹摘要、patch、固定验收、声明判定与 SACP 输出 | 达到预置预算或时间上限即停止，保留失败和不完整记录 | +| 离线复核 | 同一批轨迹上的 baseline 与 SACP 比较，可重放的证据和人工歧义裁决 | 不足以证明增量价值时报告无增益或证据不足，不发布总体优越性结论 | + +受控故障（例如错误测试选择或中断）应明确标注，与自然发生的真实失败分别报告；不能诱导虚报后当成 Agent 自然错误率。只有 fixture/mock 数据的 spike 不算真实 Agent 试点完成。 + +## 比较口径 + +先定义“固定测试退出状态＋执行范围”的基线，与 SACP 在同一批保存轨迹上比较，不能只选“完全相信最终回复”作为薄弱对照。公开任务数、运行数和 claim 数，各指标使用明确分母： + +- 任务验收成功率与不可判定比例；完成声明的支持/不支持/歧义数量。 +- 对预先定义的“不应放行”案例，SACP 与 baseline 各放行多少；报告漏检具体实例。 +- 对验收成功且满足指定证据/授权契约的案例，各误拦多少。验收成功但缺少必需证据的保守阻止单独列出，不能直接算规则 bug，也不能隐藏其使用成本。 +- 耗时、工具调用数、外层验证开销，以及 provider 实际提供的 token/费用字段。缺失费用填 unknown，不猜金额。 + +同轨迹离线比较能说明判定差异，不能证明在线接入让 Agent 任务成功率提高。若所有案例判断相同，就报告当前试点未观察到增量价值；不能事后挑选任务或修改 oracle 制造优势。 + +## 备选方案与代价 + +| 方案 | 取舍 | +|---|---| +| 只保留现有场景测试 | 快速稳定，但不能回答真实 Agent 上的漏检、误拦和接入成本 | +| 用模型运行替换所有场景测试 | 丢失确定性规则回归，给普通 CI 引入成本与波动，因此不采用 | +| 直接从多 Agent、大 benchmark 起步 | 归因与环境变量过多；首版推迟，后续按具体缺口再扩展 | +| Pi SDK 直接嵌入 | 可深度控制，但当前 Python 项目会增加 Node/TypeScript 耦合；首版选择 JSON 子进程 | +| Pi RPC 首版接入 | 适合持续会话与交互控制；首版单任务一进程优先简化，保留后续选择 | + +代价是新增 Node 运行时、模型服务依赖、轨迹隐私治理和独立验收维护。普通 PR CI 继续只运行确定性检查，已脱敏且许可允许的固定轨迹可用于以后离线回归；公开原始会话须另行批准。Pi 更换事件契约、无法隔离、或无法稳定采集时，重新审议接缝或 Agent 选择，不把 SACP 核心绑定到 Pi 私有字段。 + +## 官方依据 + +以下均为本次读取的固定源码版本，不是安装或真实执行证明： + +- [项目 README](https://github.com/earendil-works/pi/blob/b06c1f2cf1dea6af8748ddeee18c23587d2cb7c6/packages/coding-agent/README.md) 与 [package.json](https://github.com/earendil-works/pi/blob/b06c1f2cf1dea6af8748ddeee18c23587d2cb7c6/packages/coding-agent/package.json) +- [CLI](https://github.com/earendil-works/pi/blob/b06c1f2cf1dea6af8748ddeee18c23587d2cb7c6/packages/coding-agent/docs/cli.md)、[JSON events](https://github.com/earendil-works/pi/blob/b06c1f2cf1dea6af8748ddeee18c23587d2cb7c6/packages/coding-agent/docs/json.md)、[RPC](https://github.com/earendil-works/pi/blob/b06c1f2cf1dea6af8748ddeee18c23587d2cb7c6/packages/coding-agent/docs/rpc.md) +- [Security](https://github.com/earendil-works/pi/blob/b06c1f2cf1dea6af8748ddeee18c23587d2cb7c6/packages/coding-agent/docs/security.md) 与 [Environment variables](https://github.com/earendil-works/pi/blob/b06c1f2cf1dea6af8748ddeee18c23587d2cb7c6/packages/coding-agent/docs/environment-variables.md) diff --git a/docs/architecture.md b/docs/architecture.md index 09f10a8..8fc49b8 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -21,7 +21,7 @@ flowchart TB Store --> Reconciler[Caller-invoked deadline checks] Reconciler -->|Append bounded route and owner| Store end - Harness[Offline synthetic harness] -. Boundary and positive controls .-> Runtime + Harness[Scenario regression harness] -. Boundary and positive controls .-> Runtime Harness -. Strict validation .-> Validator CI[Read-only CI execution] --> Artifacts[Untrusted evidence artifacts] GitHub[GitHub API and Git objects] --> Collector[Trusted default-branch collector] @@ -71,10 +71,12 @@ The observation-only deployment health gate remains available. `can_dispatch` is ## Online engine and offline evaluation -The engine runs during a workflow to derive state and gate actions. The harness runs offline against fixed synthetic scenarios. Its eight breakers exercise missing evidence, deployment acceptance without health, transport acceptance without delivery, delivery followed by bounce, missing witness, stale approval, mock retry identity, and self-attestation. Three positive controls check that valid delivery, health and an authorized mock refund can succeed. +The engine runs during a workflow to derive state and gate actions. The existing `sacp_harness/` is a scenario regression harness for the engine: it constructs claims and mock events directly, without executing a real agent or model. Its eight breakers exercise missing evidence, deployment acceptance without health, transport acceptance without delivery, delivery followed by bounce, missing witness, stale approval, mock retry identity, and self-attestation. Three positive controls check that valid delivery, health and an authorized mock refund can succeed. Additional tests cover cross-receipt identity, callback permutations, approval revocation, SQLite restart/concurrency, and accepted-but-not-terminal reconciliation. These tests do not establish a real-world error rate. The harness's trace column is explanatory text, not a measurement of tracing systems. +[ADR-0001](./adr/0001-real-agent-evaluation-with-pi.md) accepts a separate real-agent evaluation layer, starting with Pi and a fixed model configuration. This layer is planned, not implemented. Pi is the agent harness under observation; the outer evaluation harness owns collection and independent task acceptance. LangGraph experiments remain workflow integration checks, not proof of autonomous-agent performance. The new layer will compare SACP with a test-based baseline on the same real trajectories while preserving deterministic PR CI. See [evaluation vocabulary](../GLOSSARY.md). + ## Why another layer? CI can establish that a particular job executed successfully. Tracing records the path and observations. A workflow engine persists execution progress. SACP makes the claim, required evidence, domain completion condition, authorization and next owner explicit at a handoff. If a task is already fully described by one existing CI check, another layer can add little benefit. diff --git a/docs/development.md b/docs/development.md index d2d32d0..35d767a 100644 --- a/docs/development.md +++ b/docs/development.md @@ -58,6 +58,12 @@ This last command requires a caller-supplied REST-shaped fixture; see [payload e Formal CI receipts require an independent attestation from default-branch code. Local suite JSON alone cannot create a trustworthy COMPLETE. `build_ci_receipt.py` returning zero means a valid receipt was written; it may contain BLOCK or FAIL. Consumers must read `method`, verification status, recorded SHA and current GitHub checks. See [maintainer workflow](./maintainer-workflow.md). +## Evaluation layers + +`python -m sacp_harness` remains a deterministic regression command for the SACP engine and protocol validator. It does not execute Pi, another coding agent or any model. LangGraph suites test graph/checkpointer wiring; they do not establish model-driven task performance. + +[ADR-0001](./adr/0001-real-agent-evaluation-with-pi.md) defines the planned Pi pilot: a separate Python outer runner, JSON subprocess capture, isolated/pinned runtime and independent acceptance on delivered code. No install command, agent run or new evaluation suite is implemented by that decision. Pi provider/model, runtime pin, budget and environment must be resolved before real execution. Ordinary core/protocol/harness jobs must retain their existing deterministic interface and must not silently invoke a model. + ## Public/private boundary Public docs contain reproducible commands and scoped evidence. Keep personal interview notes, outreach drafts, credentials, private logs and local absolute paths outside public commits. Historical corpus documents and saved reports retain their historical meaning; do not rewrite them to imply a new evaluation occurred.