Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .github/workflows/agentmeasure-conformance.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: agentmeasure-conformance

on:
push:
branches: [main]
pull_request:

permissions:
contents: read

jobs:
conformance:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
persist-credentials: false
- name: Check out pinned AgentMeasure without unrelated adapter dependencies
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
with:
repository: roy-tong/AgentMeasure
ref: 2cf476d6f7d0fc45401db5a822e1f12de009ac74
path: .agentmeasure-ci
persist-credentials: false
sparse-checkout-cone-mode: false
sparse-checkout: |
/action.yml
/conformance/pack/
/lab/
- name: Vector 001 - retry/cache boundary
uses: ./.agentmeasure-ci
with:
fixture: outputs/agentmeasure_urusilla_fixture_001.events.jsonl
require: execution-grain,retry-reconciliation,cost-preservation,operation-grain,evidence-boundary
- name: Vector 002 - multi-operation grain
uses: ./.agentmeasure-ci
with:
fixture: outputs/agentmeasure_urusilla_fixture_002.events.jsonl
require: execution-grain,retry-reconciliation,cost-preservation,operation-grain,evidence-boundary
11 changes: 8 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,9 +123,14 @@ protocol.
The screen's input boundary is now explicit rather than caller-asserted. A
content-derived preflight hashes exact source and transmitted bytes and records
tokenizer, chat-template, path-DAG, pre-call inclusive token-cap, and raw/JSON
success-receipt inventory. It still always blocks numeric screening because no
implemented compiler derives exact token vectors and phase bounds from those
bytes. It never chooses a session length or authorizes receiver calls. A separate
success-receipt inventory. The `/1` preflight remains inventory-only and always
blocks numeric screening. A separate
[`content-bound compiler`](initial_goal_eval/content_bound_compiler_v1.py) now
derives conservative finite token vectors from registered exact inputs and
invokes the corrected `/3` arithmetic screen. This permits only conditional
zero-call screening; no tracked real row supplies the required artifacts.
Neither layer selects a session length, authorizes receiver calls, or produces
efficiency evidence. A separate
perfect-sender diagnostic can exercise one cold Capsule comprehension followed
by `N` direct action-state turns while running matched raw and descriptive JSON
arms in fresh contexts, but its current callback accepts only a caller's
Expand Down
24 changes: 12 additions & 12 deletions decoder_qa/QA_REPORT.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Deterministic Decoder Quality-Assurance Report

- Run time (UTC): `2026-08-28T08:54:56+00:00`
- Execution ID: `d5c7caf300bd364ddfab4cdddcae16f69f65e01efadb8d51af954a87c3f1512f`
- Machine-readable result SHA-256: `06b076a343b2d109df6679b9db2e900a975fb80be651275b1516c1d9c91a2d27`
- Run time (UTC): `2026-10-01T07:55:36+00:00`
- Execution ID: `51a1f073e3aee6e7928f7ba064bafeeaef7783ab87636e4424f69a09ea3bc5d3`
- Machine-readable result SHA-256: `0e09807fc50c051c9520571ce5fad000ef788d9dd6377be152932a91c0e52daa`
- Outcome: **passed**
- Scope: local saved parser fixtures and documented grammar only
- External activity: none; network audit events were denied and credential environment variables were not inherited
Expand All @@ -17,13 +17,13 @@ This is quality-assurance evidence, not a conformance badge, security audit, vul

| Campaign | Status | Cases/tests | Seconds |
|---|---:|---:|---:|
| baseline | passed | 101 | 2.785780 |
| roundtrip | passed | 1097 | 0.652272 |
| boundaries | passed | 1952 | 0.262333 |
| mutations | passed | 2048 | 0.152693 |
| replay | passed | 135 | 0.524518 |
| known_defects | passed | 16 | 0.084606 |
| qa_tests | passed | 15 | 1.601346 |
| baseline | passed | 101 | 23.138659 |
| roundtrip | passed | 1097 | 2.937603 |
| boundaries | passed | 1952 | 1.268036 |
| mutations | passed | 2048 | 0.852674 |
| replay | passed | 135 | 2.657946 |
| known_defects | passed | 16 | 0.513651 |
| qa_tests | passed | 15 | 5.852328 |

## Exact behavior check-unit counts

Expand Down Expand Up @@ -133,8 +133,8 @@ These local results show that the saved implementation rejects or handles the ex
## Source identity and stability

- Source files hashed: `37`
- Source snapshot digest: `df3613689a7020611b296592362d722ec7b821415879f32ecf5c478cbb9b7b82`
- Git revision: `9cc2dfd0526422e8663499cdd9aa2b25da91ad8a`
- Source snapshot digest: `83a15d338e3875d5bfc3dad61926b74f30e7d306bfedcd13a9e16b3ed17558f9`
- Git revision: `f97adf6d0dd929f3e28a8549abf1889b5ba4948e`
- Grammar Capsule SHA-256: `588034f997fb4f3d35dfdbb68afd9232a78192ac1fa497d565f67e0892358a27`
- Capsule-pinned reference codec SHA-256: `3cb84380149a522bdadd94e866e39a848fa23c8b666382c4f88ce341147fbdcf`
- Saved reference codec SHA-256: `3cb84380149a522bdadd94e866e39a848fa23c8b666382c4f88ce341147fbdcf`
Expand Down
6 changes: 3 additions & 3 deletions decoder_qa/SHA256SUMS
Original file line number Diff line number Diff line change
@@ -1,17 +1,17 @@
244d332dd991eeb168902059153e709cb545f7cd8410d2a1f32ce4b1974965a9 AGENTS.md
9bb38dcdef72ee0207a0350ca17896fca01e8ee3d0ecf5a145dcf96bc55ab242 GOVERNANCE.md
5c5c0df2c5104ea682ea72344b33b3b743b5f9a4693e6c6132ca74e64df28a78 PROVENANCE.md
d220bd9c218c86faab3bca0980af3aa999e5882bb6cbf110346634c1a0da2dc6 README.md
297d35316a329bdc44b851e03e9be488517515a8e66700611857b87080de236c README.md
3ab77aa1d0d67982d0762979aa09b36216bc46124cd847089ed4e6da94f97da9 RESEARCH_PROGRAM.md
927515598e98302da6a715e0b1a02f52c278b798dce3a3a405e37242265a53a2 SECURITY.md
95042c6e31d79e213f135ab8ea275eab8081d450432cf8a1a1979d78d53f4252 SOURCE_MANIFEST_FORMAT.md
e04d0f01639dbb35782f6bf0b8eb4a48f6527822ed2b183e461fcc7740ee5a14 URUSILLA_INTERNET_LAYER.md
f1d85bb976da3e4c7765d7cce0ce320a8dd53adc4e3db54078deeb827817f65a decoder_qa/QA_REPORT.md
2190128b75e990e9cd57d7bae66779c667fba2c3137af3471f3d472458b3a722 decoder_qa/QA_REPORT.md
bbb13bf079da7f5700362e13c83a4828dfc9ce8e539f397bee309c66d5aa20a4 decoder_qa/README.md
2d4d936913ccc6ce321d6b09f24b3fec97b33c0fdce40ceadc3ddca01a27b072 decoder_qa/__init__.py
68f891d8d493a3f907ad8fbcd5c62ad7d7f026eb99ca4ed8185437121ed5c695 decoder_qa/known_failures.json
18d3540ed877299e5e171eb3808b5251ebac60577dd9cd620fff13d84aebaca6 decoder_qa/qa_core.py
06b076a343b2d109df6679b9db2e900a975fb80be651275b1516c1d9c91a2d27 decoder_qa/qa_results.json
0e09807fc50c051c9520571ce5fad000ef788d9dd6377be152932a91c0e52daa decoder_qa/qa_results.json
8383c65d42c9db6b19ba7c5a22636d0bf62cc30d39a1dfaeb670de74a64ca925 decoder_qa/qa_worker.py
c09d548aae72d7f001c90d35930c2441dc9c74ba174c603eee1d7c3039aa6024 decoder_qa/run_decoder_qa.py
655a04326b71b4b0f037dfa251d3bb2cbb044969759b88423733ae63715a68c1 decoder_qa/test_decoder_qa.py
Expand Down
42 changes: 21 additions & 21 deletions decoder_qa/qa_results.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 6 additions & 3 deletions urusilla_hybrid_runtime/session.py
Original file line number Diff line number Diff line change
Expand Up @@ -955,6 +955,12 @@ def _execute(
"adapter-reply-type-invalid",
"session adapter returned an invalid reply type",
)
# A discarded reply still reports provider usage for this call.
# Record it before context, receipt, or budget checks invalidate
# the session; responses without a valid reply remain unreported.
self._turn_provider_reported_tokens.append(
reply.reply.provider_total_tokens
)
if reply.context_reset_observed:
self._fail("context-reset", "provider reported a context reset")
if reply.context_compaction_observed:
Expand Down Expand Up @@ -1029,9 +1035,6 @@ def _execute(
)
self._last_provider_receipts_sha256 = reply.receipts.sha256
self._next_turn += 1
self._turn_provider_reported_tokens.append(
reply.reply.provider_total_tokens
)
return SessionTurnResult(
lease_sha256=lease.sha256,
response_sha256=reply.receipts.response_content_sha256,
Expand Down
13 changes: 12 additions & 1 deletion urusilla_hybrid_runtime/tests/test_session.py
Original file line number Diff line number Diff line change
Expand Up @@ -461,6 +461,11 @@ def test_provider_reset_or_compaction_reply_invalidates(self) -> None:
expected_reason,
)

self.assertEqual(
session.snapshot().cost.turn_provider_reported_tokens,
(15,),
)

def test_reply_binding_and_receipt_mismatches_invalidate(self) -> None:
cases = (
(
Expand Down Expand Up @@ -530,7 +535,7 @@ def test_reply_binding_and_receipt_mismatches_invalidate(self) -> None:
)

def test_token_ceiling_and_adapter_failure_are_terminal(self) -> None:
session, _, _ = new_session()
session, attempt, _ = new_session()
lease = prepare_session_turn(
session,
"bounded",
Expand All @@ -540,6 +545,11 @@ def test_token_ceiling_and_adapter_failure_are_terminal(self) -> None:
with self.assertRaises(SessionError):
execute_session_turn(session, lease, SessionAdapter())
self.assertEqual(session.invalidation_reason, "token-budget-exceeded")
self.assertEqual(session.snapshot().cost.turn_provider_reported_tokens, (15,))
self.assertEqual(
session.snapshot().cost.reported_call_total_tokens,
attempt.evidence.provider_total_tokens + 15,
)

class FailingAdapter:
def complete_session_turn(self, handle, call):
Expand All @@ -555,3 +565,4 @@ def complete_session_turn(self, handle, call):
with self.assertRaises(SessionError):
execute_session_turn(session, lease, FailingAdapter())
self.assertEqual(session.invalidation_reason, "adapter-call-failed")
self.assertEqual(session.snapshot().cost.turn_provider_reported_tokens, ())
4 changes: 3 additions & 1 deletion website/app/api/project-activity/route.ts
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,9 @@ export async function GET(request: Request) {
const lastGoodCacheKey = new Request(new URL('/api/project-activity?cache=last-good-v1', request.url), {
method: 'GET',
});
const edgeCache = typeof caches === 'undefined' ? null : caches.default;
const edgeCache = typeof caches === 'undefined' || !('default' in caches)
? null
: (caches as CacheStorage & { default: Cache }).default;
if (edgeCache) {
try {
const cached = await edgeCache.match(cacheKey);
Expand Down
3 changes: 2 additions & 1 deletion website/app/live-project-desk.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ type DeskData = {

type DeskSnapshot = DeskData & {
generated_at?: string;
status?: 'stale';
};

type CachedDesk = {
Expand Down Expand Up @@ -213,7 +214,7 @@ export function LiveProjectDesk() {
setData(nextData);
setCheckedAt(receivedAt);
writeCachedDesk(nextData, receivedAt);
setStatus('ready');
setStatus(live.status === 'stale' ? 'error' : 'ready');
} catch {
setStatus('error');
} finally {
Expand Down
Loading
Loading