Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Positive controls passed: 3/3

This is a **scenario regression harness for the SACP engine**: it constructs claims and mock events directly, without running a coding agent or calling a model. It prints by default; `--output-dir .local/harness` explicitly writes reports. Continue through the [example guide](./examples/README.md) for deployment, refund, recovery and optional LangGraph demos.

A separate real-agent evaluation layer is planned, with Pi as the first integration pilot. It is not implemented or benchmarked yet; see [ADR-0001](./docs/adr/0001-real-agent-evaluation-with-pi.md) for the decision, independent acceptance criteria and limits.
The separate **Pi integration** now provides a no-model installation probe, JSON event replay and independent source-test acceptance. Try `python -m examples.pi_evaluation`: a scripted success claim passes for correct code and is blocked for incorrect code. This is not a real-agent benchmark; see the [integration guide](./docs/pi-integration.md) and [ADR-0001](./docs/adr/0001-real-agent-evaluation-with-pi.md).

<a id="architecture"></a>

Expand Down
2 changes: 1 addition & 1 deletion README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ Positive controls passed: 3/3

这里是 **SACP 引擎的场景回归测试(Harness)**:直接构造声明与模拟事件,不运行 coding agent,也不调用模型。默认只打印,显式传入 `--output-dir .local/harness` 才写报告。接着按[示例导航](./examples/README.zh-CN.md)体验部署、退款、恢复和可选 LangGraph demo。

另计划新增真实 Agent 评测层,以 Pi 为首个接入试点。目前尚未实现,也没有对应 benchmark 结果;选择理由、独立验收标准与边界见 [ADR-0001](./docs/adr/0001-real-agent-evaluation-with-pi.md)。
独立的 **Pi 接入层**已提供无模型安装探测、JSON 事件回放和交付源码的独立测试验收。运行 `python -m examples.pi_evaluation`:同一条合成成功声明,正确源码通过、错误源码被阻止。尚无真实 Agent benchmark;用法与边界见[接入指南](./docs/pi-integration.md)和 [ADR-0001](./docs/adr/0001-real-agent-evaluation-with-pi.md)。

<a id="框架"></a>

Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ CI runs a cross-platform core matrix, explicit LangGraph integration, strict pro

## Next: real-agent evaluation pilot

[ADR-0001](./docs/adr/0001-real-agent-evaluation-with-pi.md) retains the existing deterministic engine scenarios and selects Pi as the first real-agent pilot. Implement a Python outer runner using Pi's JSON subprocess interface, isolate and pin its environment, then collect actual coding-task trajectories and independently verify the delivered patches. The adapter, provider/model, budget and real results are not yet available.
[ADR-0001](./docs/adr/0001-real-agent-evaluation-with-pi.md) retains deterministic engine scenarios and selects Pi as the first real-agent pilot. The [initial integration](./docs/pi-integration.md) provides a no-model RPC probe, JSON subprocess capture API with default-denied model execution, event replay and independent acceptance of explicit Python fixture files. Next enforce isolation, pin the complete environment, select a provider/model and budget, then collect actual coding-task trajectories and verify delivered patches. No real-agent task result is available yet.

Start with a small fixed task set and compare SACP with a predefined test-based baseline on the same records. Report missed false completions, incorrect blocks, indeterminate outcomes and overhead; separate injected faults from naturally occurring failures. Keep model calls out of required PR CI. Acceptance stages and stop conditions are defined in the ADR.

Expand Down
2 changes: 1 addition & 1 deletion ROADMAP.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ CI 已运行跨平台 core 矩阵、显式安装依赖的 LangGraph 集成、严

## 下一步:真实 Agent 评测试点

[ADR-0001](./docs/adr/0001-real-agent-evaluation-with-pi.md)决定保留现有确定性引擎场景测试,以 Pi 为首个真实 Agent 试点。计划使用 Python 外层 runner 与 Pi JSON 子进程接口,固定并隔离环境,采集实际代码任务轨迹并独立验收交付补丁。目前尚无 adapter、选定的 provider/model、运行预算或真实评测结果。
[ADR-0001](./docs/adr/0001-real-agent-evaluation-with-pi.md)决定保留现有确定性引擎场景测试,以 Pi 为首个真实 Agent 试点。[初版接入](./docs/pi-integration.md)已有无模型 RPC 探测、默认禁止模型执行的 JSON 子进程采集 API、事件回放及明确列出的 Python fixture 文件独立验收。下一步落实强制隔离、完整环境锁定、provider/model 与预算,再采集实际代码任务轨迹并验收补丁。目前没有真实 Agent 任务结果。

先用小规模固定任务集,在同一批记录上比较 SACP 与预先定义的测试基线,报告漏检、误拦、不可判定和额外开销;注入故障与自然失败分别统计。模型调用不进入 required PR CI。分阶段验收与停止条件见 ADR。

Expand Down
5 changes: 3 additions & 2 deletions STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,13 @@ Start with [README](./README.md) / [中文](./README.zh-CN.md), then [architectu
| Adapters | REST-shaped GitHub payload mapping, trusted reviewer allowlist, CLI, real LangGraph wiring/checkpointers | Payload source authentication remains an integration responsibility |
| HTTP experiments | Real localhost provider transport, simulated health/rollback | No public SACP API or production provider |
| Scenario regression harness | Eight synthetic breakers and three positive controls for the SACP engine | Does not run a real agent/model; no measured population accuracy |
| Pi evaluation integration | No-model RPC probe, default-denied JSON capture API, replay and independent Python fixture acceptance | No model task run, authenticated collector or hostile-code sandbox; see [guide](./docs/pi-integration.md) |
| Governance | Read-only PR CI, trusted independent attestation, formal receipt, manual squash merge, main revalidation | Semantic review required; real fork E2E still outstanding |
| Corpus and Doctor | 34 translated receipts, YAML examples, local text-audit CLI | Historical/synthetic inputs; Doctor uses heuristics and does not rerun tasks |

## Verification baseline

Code baseline [1b76ff3](https://github.com/aDragon0707/sacp/commit/1b76ff386d6091535aae94acbb7517c9170bd5d8): 100 core tests, 7 LangGraph tests, 49 strict protocol documents, 8 breakers and 3 positive controls. Core CI covers Ubuntu Python 3.11/3.12 and Windows Python 3.12; LangGraph runs on Ubuntu Python 3.12. Zero failures/errors/unexpected skips are required.
Historical code baseline [1b76ff3](https://github.com/aDragon0707/sacp/commit/1b76ff386d6091535aae94acbb7517c9170bd5d8): 100 core tests, 7 LangGraph tests, 49 strict protocol documents, 8 breakers and 3 positive controls. The Pi integration increases the core inventory to 136, including 35 no-model Pi tests and one classification test. Local execution on 2026-10-15 passed all four suites with no failures/errors/skips; SHA-bound CI artifacts remain the evidence for each merged revision. Core CI covers Ubuntu Python 3.11/3.12 and Windows Python 3.12; LangGraph runs on Ubuntu Python 3.12.

See the [dated verification record](./docs/verification-2026-10-15.md) for exact runs and smoke coverage. A successful receipt-generation job can write a BLOCK or FAIL receipt; inspect content and current SHA, not just a green job.

Expand All @@ -33,6 +34,6 @@ The protocol itself does not require a database or HTTP transport. The Python re

## Next work

The accepted [real-agent evaluation decision](./docs/adr/0001-real-agent-evaluation-with-pi.md) preserves deterministic scenarios and selects Pi as the first coding-agent pilot, using an outer runner and independent task acceptance. This is planned work: no Pi adapter, selected provider/model or real-agent benchmark result exists yet.
The accepted [real-agent evaluation decision](./docs/adr/0001-real-agent-evaluation-with-pi.md) now has an [initial no-model integration](./docs/pi-integration.md). Actual Pi 1.0.0 `get_state` passed without prompts; deterministic replay and independent source acceptance exercise the evidence path. A selected provider/model, enforced execution isolation, budget and real-agent benchmark results remain future work.

Finish controlled real fork end-to-end acceptance, strengthen isolation against arbitrary malicious Python tests, authenticate external collectors where used, and gather independently reviewed real-world false-completion and false-block cases. Keep schema/transport expansion behind concrete dirty cases and compatibility review. See [ROADMAP](./ROADMAP.md).
8 changes: 7 additions & 1 deletion docs/adr/0001-real-agent-evaluation-with-pi.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ date: 2026-10-15

当前 `sacp_harness/` 直接构造 Claim、mock provider 事件和授权,验证 SACP 引擎及协议文档;没有驱动真实 Agent 或调用模型。我们决定保留它作为确定性场景回归测试,另增真实 Agent 端到端评测层,并选择 Pi coding agent 作为首个有界接入试点,以回答 SACP 相比已有测试检查增加了什么判别价值。

本 ADR 接受的是架构方向和首个试点对象。Pi adapter、执行环境、任务集和评测结果均尚未实现;源码可接入性已核对,安装兼容性、隔离和实际采集须经后续 spike 验收。
本 ADR 最初接受的是架构方向和首个试点对象;当时 Pi adapter、执行环境、任务集和评测结果均尚未实现。后续进展独立记录于文末,保留以下原始决策与源码证据边界。

## 决策与术语

Expand Down Expand Up @@ -35,6 +35,12 @@ date: 2026-10-15

选择 Pi 的理由是可观察和易接入,不是已有证据表明它的任务成功率优于其他 Agent。

### 实施进展:2026-10-15,无模型接入阶段

新增 `sacp_eval/`:JSON 事件解析、默认拒绝模型执行的子进程采集 API、独立源码 fixture 验收与 SACP 证据绑定。本机已安装的 Pi 1.0.0 通过 `--version` / RPC `get_state` 探测,未发送 prompt、未继承模型凭据;这不代表原始审查的 1.2.0 版本已安装或完整运行时已锁定。RPC 此处只用于状态探测,任务执行接缝仍为 JSON subprocess。

合成轨迹与真实测试子进程已能区分正确/错误交付,并阻止旧源码证据复用;没有真实模型轨迹或效果指标。强制隔离、provider/model、预算和完整环境锁定尚未落实。用户明确要求先完成接入、暂不调用模型,因此完整真实 Agent pilot 仍待后续授权和环境准备。用法、接口与信任边界见[Pi 接入指南](../pi-integration.md)。

## 边界与执行流程

```mermaid
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ The engine runs during a workflow to derive state and gate actions. The existing

Additional tests cover cross-receipt identity, callback permutations, approval revocation, SQLite restart/concurrency, and accepted-but-not-terminal reconciliation. These tests do not establish a real-world error rate. The harness's trace column is explanatory text, not a measurement of tracing systems.

[ADR-0001](./adr/0001-real-agent-evaluation-with-pi.md) accepts a separate real-agent evaluation layer, starting with Pi and a fixed model configuration. This layer is planned, not implemented. Pi is the agent harness under observation; the outer evaluation harness owns collection and independent task acceptance. LangGraph experiments remain workflow integration checks, not proof of autonomous-agent performance. The new layer will compare SACP with a test-based baseline on the same real trajectories while preserving deterministic PR CI. See [evaluation vocabulary](../GLOSSARY.md).
[ADR-0001](./adr/0001-real-agent-evaluation-with-pi.md) accepts a separate real-agent evaluation layer. Its initial `sacp_eval/` implementation includes a no-model Pi probe, default-denied subprocess capture seam, JSON lifecycle parser and independent acceptance of explicit Python files. The trusted outer caller binds trace and delivery digests before recording a host test observation and selected claim. Pi is the agent harness under observation; the outer evaluator owns collection and acceptance. Current demos replay scripted events; no real model-driven task has run. Real trajectories, isolated execution and baseline comparison remain future work. LangGraph experiments remain workflow wiring checks. See the [integration guide](./pi-integration.md) and [evaluation vocabulary](../GLOSSARY.md).

## Why another layer?

Expand Down
4 changes: 2 additions & 2 deletions docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ python -m unittest tests.test_harness -v
git diff --check
```

The code baseline linked from [README](../README.md) has 100 core tests, 7 integration tests, 49 protocol documents, 8 breakers and 3 positive controls. Counts are a versioned inventory, not a fixed success criterion. New tests should increase coverage; a missing dependency or skip is not a complete pass.
The historical code baseline linked from [README](../README.md) has 100 core tests, 7 integration tests, 49 protocol documents, 8 breakers and 3 positive controls. The Pi integration increases core discovery to 136 tests. Counts are a versioned inventory, not a fixed success criterion. New tests should increase coverage; a missing dependency or skip is not a complete pass.

The core suite discovers `tests/test_*.py`, excluding the optional LangGraph and harness modules. `--suite harness` executes the breaker/positive report directly; `tests.test_harness` separately checks report output and deterministic artifact handling. Protocol validates root YAML examples and translated corpus receipts strictly.

Expand Down Expand Up @@ -62,7 +62,7 @@ Formal CI receipts require an independent attestation from default-branch code.

`python -m sacp_harness` remains a deterministic regression command for the SACP engine and protocol validator. It does not execute Pi, another coding agent or any model. LangGraph suites test graph/checkpointer wiring; they do not establish model-driven task performance.

[ADR-0001](./adr/0001-real-agent-evaluation-with-pi.md) defines the planned Pi pilot: a separate Python outer runner, JSON subprocess capture, isolated/pinned runtime and independent acceptance on delivered code. No install command, agent run or new evaluation suite is implemented by that decision. Pi provider/model, runtime pin, budget and environment must be resolved before real execution. Ordinary core/protocol/harness jobs must retain their existing deterministic interface and must not silently invoke a model.
[ADR-0001](./adr/0001-real-agent-evaluation-with-pi.md) defines the Pi pilot. The [initial integration](./pi-integration.md) now supports `python -m examples.pi_evaluation` and `python -m unittest tests.test_pi_evaluation -v`, without Pi installation, model calls or credentials. Core discovery includes these tests. Only the opt-in `probe` command uses an installed Pi, for version and RPC state. Real model runs still require a selected provider/model, budget, complete runtime pin and enforced isolation; the capture library defaults to refusing model execution. Local acceptance is for trusted fixtures only. Ordinary required CI retains its deterministic interface.

## Public/private boundary

Expand Down
1 change: 1 addition & 0 deletions docs/maintainer-workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ The policy checker records zero or more classes in CI evidence and the formal re
|---|---|---|
| `protocol` | protocol documents and YAML packets | semantics, compatibility, dirty case, example, changelog |
| `engine` | `sacp_verify/`, `sacp_harness/` | conservative state projection, append-only history, failure tests |
| `adapter` | `sacp_eval/`, Pi evaluation examples | event contract, delivery binding, independent acceptance and explicit execution boundaries |
| `adapter` | GitHub, LangGraph, and CLI boundaries | external facts must not be promoted to stronger states |
| `automation` | workflows and CI scripts | permissions, action pins, self-modification, evidence binding |
| `corpus` | `sample-corpus/` | privacy, contamination, reproducibility |
Expand Down
Loading
Loading