Skip to content

Latest commit

 

History

History
272 lines (229 loc) · 23.2 KB

README.md

File metadata and controls

272 lines (229 loc) · 23.2 KB

Delegation contract templates

Starting points for local-offload delegate --contract <file> and the MCP agent_delegate tool (each template is one subtask object; the same shape works inside an agent_delegate subtasks array, and a file may also hold an array of up to 8). Copy one, replace the context_paths placeholders with real files under your --read-root, and sharpen the goal. Enable recipe and a worked run: docs/OPERATOR-GUIDE.md; wire semantics: docs/FLEET-NODE.md.

JSON carries no comments, so the field guidance lives here.

Fields (the --contract file shape)

Field Required Notes
goal yes Self-contained — the sub-agent sees only this plus the context docs. Name the output you want and what "wrong" looks like; a vague goal is the number-one cause of schema-valid garbage.
context_paths no Files the delegator reads and inlines as context docs (≤ 128 KiB each), confined to --read-root. The templates ship placeholders — replace them.
context no Inline docs [{name, text}] if you already hold the text; ≤ 16 docs, ≤ 256 KiB total including anything context_paths adds.
output_schema for remote JSON Schema with a properties map — string / number / integer / boolean / enum fields, lists of strings, and objects and lists of objects nested to any depth (the grammar-compilable subset). A list of numbers, booleans or enum values is written by the grammar as a list of quoted strings and converted back to the schema's type; a type list, anyOf, oneOf or $ref reads as a string; every property the schema lists is required by the grammar, at every depth. Required for any remote placement; keep required listed, the validator enforces it.
acceptance recommended Delegator-evaluated DSL (below). A result failing any check is failed_verification, never a success.
profile no Default research (read-over-docs).
max_steps no Default 12, clamped to 12.
timeout_sec no Default 300, clamped to 900. Size it to the seat — a weak seat reading three docs can legitimately need minutes.
write_root no Opens the WRITE door (0.122.0, D-06): a directory RELATIVE to the run's read root that the seat may create and change files under. The executing node must have agent_allow_write: true or the contract is refused (ack 400 on the fleet path, defer_class: "write" locally). The seat works on the node's own throwaway copy of the docs and the result carries a unified diff + diff_files the caller applies — the harness never does. Grants create+overwrite only: no delete, no shell, no run, no network. Caps: 8 files, 64 KiB written, 192 KiB of diff; past any of them nothing is published.

schema_version and depth are minted by the delegator; putting them in a file has no effect.

Acceptance DSL

Check Passes when Use for
contains:<s> output contains s a term the answer must mention
not_contains:<s> output lacks s banned filler (TODO, an apology phrase)
regex:<re> Go regexp matches output shape demands, e.g. regex:[0-9] = "carries a number"
min_items:<field>:<n> structured.<field> is an array with ≥ n items (n ≥ 1) minimum yield from an extraction
nonempty:<field> structured.<field> present and non-empty (0/false count as values) required fields that must not be omitted
nonempty:<f1>|<f2>|… at least ONE of the named fields is present and non-empty (same rules; an absent field counts as empty) "the answer said something": a digest whose every field is empty fails, one with a single value passes
diff_touches:<prefix> the write set holds a changed path starting with prefix a write_root contract: the leg changed the file it was pointed at
diff_max_files:<n> the write set is NON-EMPTY and touches ≤ n files a write_root contract: the leg stayed in its lane

Text verbs read the final output, falling back to the raw structured bytes when output is empty; the field verbs require structured and fail closed without it. Unfalsifiable checks (contains:, min_items:f:0) are rejected at validation.

Authority-graded checks (optional qualifier)

A check says what must hold; an authority says how far the evidence behind it can be trusted. contains:, regex: and not_contains: take an optional qualifier at the head of the argument. It is not a verb: the verb set above is closed and unchanged, an unqualified check behaves exactly as it always did, and a node one release behind reads the line as a plain substring (a node parses acceptance when it ACKs a contract and never evaluates it).

Check Reads Passes when
contains:@answer:<s> the answer text the model wrote (output only: no fall back to structured) s is in it
contains:@field=<name>:<s> the values inside structured field <name> (text, numbers and booleans, never keys) s is in them
contains:@grounded:<s> the answer text and the values in structured, against the contract's context docs s is in the answer AND verbatim in at least one context doc
regex:@answer:<re>, regex:@field=<name>:<re> as above the pattern matches
regex:@grounded:<re> as above a non-empty match in the answer is verbatim in at least one context doc
not_contains:@answer:<s>, not_contains:@field=<name>:<s> as above s is absent from it. There is no grounded form: documents cannot corroborate an absence.

The levels are ordered, weakest first: answer (what the model wrote; only the model vouches for it), field (a value in the structured result, a slot the contract's own schema asked for) and grounded (text the answer carries that the delegator's own documents also carry: a corroborated mention, not proof the claim is true). Grounding compares the matched text verbatim and case-sensitively, so a case-insensitive pattern can match the answer and still fail the document check. Three rules follow.

  1. Absent evidence is unknown, never a pass. An empty answer, no structured result, an absent or empty field, no answer at all, or no context docs for a grounded check leaves the check with nothing to read. It grades unknown: the merge gate counts it as a failed check, and the reason says unknown at <level> authority: ... so an empty answer is told apart from a wrong one. not_contains:@answer:<s> over an empty answer is therefore unknown, where the unqualified not_contains:<s> passes.
  2. A weaker authority never closes a stronger claim. Each level reads exactly the evidence it names. An answer that says walrus closes contains:@answer:walrus; it does not close contains:@field=summary:walrus (no field carries it: unknown or fail) and it does not close contains:@grounded:walrus (no document vouches for it: fail, or unknown when the contract has no documents). core.ClosedAt (and delegate.GradeAcceptance, the per-check outcomes) report the authority a contract closed at: the weakest among its checks, and only when all passed, so one unqualified check pulls it to none. The delegator publishes it as results[].acceptance_authority (answer, field or grounded) and on the corpus row, only for a result accepted whole: with an unqualified check, a failed or unknown check, a list-fidelity failure or an excused document fingerprint the key is absent, and the result publishes as it always did.
  3. Unqualified checks are untouched. In particular not_contains:<s> over a fully empty result still passes. That is the vacuous pass the qualifier avoids. It stays for the unqualified form because changing it would change the verdict of contracts this repository cannot see, in a closed DSL every contract uses, and it is not shown unreachable: an empty final answer is deferred before acceptance, so the case is rare, but results also arrive from nodes, the queue and the rescue path. No file under contracts/ (the write-door contracts included) uses not_contains:, and the operator guide's example pairs it with min_items: and nonempty:, which already fail closed on an empty result. A contract that cannot afford the vacuous pass writes the qualified form, or pairs the not_contains: with a qualified contains: on the same evidence.

A field-level check names its field, so the field joins the schema's required list the way nonempty: and min_items: fields do. The intake lint adds AUTHORITY-UNREACHABLE (results[].acceptance_lint, warn-only): a @grounded: check on a contract with no context docs, and an @field= check whose field the output_schema does not declare (or that has no output_schema), can only ever grade unknown (a blank document counts as none, as it does when grading). Such an unknown is a defect of the contract, not of the node that answered. On the research lane's document fingerprint (the wrong-document tripwire a node is quarantined for) it is still reported as a failed check, but it is neither a strike nor a pass for the node, and the contract is not retried on a second node, because no node's answer could close it. The same holds for an unknown over an honest-absence digest. Only a FAILED fingerprint can be excused (by an honest absence). Apart from those two cases, an unknown the contract could have closed (an empty answer beside the documents, a declared field the result lacks) is the result's, and strikes like a failure.

Spelling: only @answer:, @grounded: and @field=<name>: (on contains:, regex: and not_contains:) are read as qualifiers. The reserved prefixes are exactly @answer:, @grounded:, @field= and @field: (the last is no qualifier: it is refused, because the field qualifier's separator is =). After a reserved prefix a missing field name (@field=:x), a missing colon after the name (@field=summary), an empty tail, or @field: itself is a parse error, and so is not_contains:@grounded:; lines like these were plain substrings before. A substring that starts with a reserved prefix is now read as a qualified check, so anchor on something else or write the first character as a class (regex:[@]answer:). Text that only begins with the letters @field is not a reserved prefix, so contains:@field, contains:@fieldname:x and contains:@fieldx are plain literal substrings, as is any other leading @, as before. A misspelt qualifier, and a qualified line read by a delegator one release behind, are a literal substring: for contains: and regex: no answer carries it, so the check fails closed. For not_contains: that delegator reads not_contains:@answer:SECRET as the literal string "@answer:SECRET", so the check passes even when the answer contains SECRET and the forbidden-content check never fires. Qualified checks need the whole fleet on 0.187 or later.

Acceptance carries no wall-time term (INV-5, ADR 0050: "Acceptance criteria and pass rules stay wall-time-free"). No verb above reads a clock: each reads the answer text, a structured field or the write set, so a time criterion cannot be written as a check and a result is never judged by how fast it arrived. Two things could still put a speed term into an acceptance string: a content check that quotes a speed word (contains:faster, regex:within 90 s, regex:tok/s >= 20) and a future verb that reads a clock. TestContractAcceptanceCarriesNoSpeedTerm refuses the first in every JSON file under this directory, nested ones included (the key acceptance is read in any case, as the contract reader matches it: Acceptance is a live list), and TestAcceptanceVerbSetIsReviewed makes the second a reviewed decision: the verbs the parser declares and accepts must equal a reviewed list that gives each verb the reason it reads no clock. The scan is a lint, not a proof. It reads a regex: argument with its escapes undone, but a figure spelled \d+, [0-9]+ or (?:a|b) evades it, so the closed verb set is the defence that does not depend on how a check is spelled. A check that has to quote a doc fact which contains such a phrase anchors on the figure instead (regex:90), as every content anchor below does.

The any-of form of nonempty: uses the same emptiness rules as the one-field form, and a failure is one reason that names the check and says why each field is empty or absent. An empty alternative (a||b, |a, a|) and a repeated name (a|a) are rejected at validation, names are exact (no trimming), and a single name is the one-field check unchanged. Only nonempty: alternates: min_items: takes its field name literally. A node parses a contract's acceptance when it ACKs it and never evaluates it, so a node one release behind admits a contract that carries the any-of form.

The two diff verbs read the run's WRITE SET and are the only checks a talkative seat cannot satisfy by talking. Both fail closed on an empty write set, diff_max_files included: a cap assertion that passed because nothing was written would make a contract that verified nothing read as verified. They do not check that the change is CORRECT — nothing mechanical can. Read the diff.

Authoring rule: anchor at least one check to content that appears only in the docs

Acceptance is what makes a result verified and what fires the cross-seat retry — and three authoring shapes, each measured in the standing corpus, quietly disable it. The intake lints every contract and returns warnings per subtask (results[].acceptance_lint, warn-only):

  • PARROT-PASSABLE — every content check is also satisfied by the goal text itself, so a model that echoes the question back passes as verified and the retry never fires (measured on 5/5 of the first organic contracts). Note not_contains:<s> counts as parrot-passable when s is absent from the goal — an echoed question trivially lacks it.
  • UNGROUNDED — a contains:/regex: matching nothing in the contract's own context docs fails RIGHT answers (measured: a contains: check on a name the context docs never wrote failed both seats on a task both did right).
  • SHAPE-ONLY — nonempty:/min_items: alone verify that fields exist, not that they are true (measured: "the docs directory does not exist" passed nonempty:summary).

The fix is one habit: pick a term or figure that appears in the docs but not in the goal (a number, a proper noun the question does not name) and anchor a contains:/regex: to it.

The templates

Template Shape Acceptance logic
docs-drift-scan.json one doc vs. source excerpts → drifted claims + verdict verdict and count must exist; drifted may legitimately be empty (a clean doc), so no min_items on it
bench-log-digest.json one benchmark log → best config, regressions, summary a digest with no digit anywhere is wrong: regex:[0-9]
schema-extraction.json one reference doc → the identifiers it documents an extraction yielding zero keys is a failure: min_items:keys:1
research-digest.json several docs → grounded findings, open questions, summary a synthesis is only a synthesis with ≥ 3 findings: min_items:findings:3

Run one

local-offload delegate --contract contracts/research-digest.json --read-root . --route auto --remote http://<node-b>:18811

Read the response's summary block first (succeeded / deferred / failed_verification / failed / infrastructure); exit 0 covers honest defers and failed verification, non-zero is transport/config failures and infrastructure > 0 — the defers whose defer_class blames a broken or misconfigured node rather than the work.

write-door/ — the write door's four-task gate fixtures (registers D-06, D-114)

Four staged implementation legs, one directory each with its own contract.json (write_root: ".", thinking: "off", diff-verb acceptance): t1 a one-file Go fix (Clamp returns the wrong bound — go test is RED until it is fixed), t2 a two-file Go fix plus one table case (ParsePort never rejects > 65535 — the fix and the case that exercises it), t3 a JSON + Markdown record edit (flip one node's flag and its table cell, nothing else), t4 a ~3 KB whole-file rewrite asked for in ONE write_file call (the shape whose cut argument the engine refuses — register D-114 — proved by comparing the applied copy line by line against the fixture: exactly the status line may differ). scripts/write-door-gate.ps1 sends them to one seat, applies each returned diff to a FRESH copy and proves it there (go test, a JSON parse, exact-row checks) — the proof a caller of any write contract owes, since the harness never applies its own writes. Run it after opening agent_allow_write on a node:

scripts/write-door-gate.ps1 -Remote http://<node>:18811     # a fleet node's door
scripts/write-door-gate.ps1                                  # this box's own seat, route local

Measured 2026-09-15: 3/3 on a 4B vLLM seat (18 / 45 / 48 s) and t1 in 10 s on a 27B seat; t4 is the shape measured the same day on the <node-a> 9B llama.cpp seat, where a 1,024-token step budget cut the write mid-argument. The files are pinned eol=lf in .gitattributes so a seat's LF diff applies on every checkout.

digest-8.json — the parallel-sessions gate fixture (0.111.0)

Eight self-contained digest subtasks (inline context: this repo's own ADRs), used by scripts/parallel-sessions-gate.ps1 to run K concurrent 8-wide fan-outs against one llama-swap and prove the seat-contention wait fires instead of deferring (ADR 0032). Needs no --read-root.

digest-adr-hard-8.json — the HARDER set: three ADRs per contract, four anchored findings per document (register G-36)

The standard instrument (INV-6) SATURATED on digest-8.json: every 27B-class arm scored accuracy ~9 on 2026-09-18 and the top pair came back TIE, which the instrument names a measurement failure to fix by rebuilding the set, never by ranking on speed. This fixture is that rebuild. Each of its eight contracts inlines THREE ADRs (24 ADRs, none of the eight digest-8.json uses and never 0023 or 0029, whose text the pre-push leak scan refuses; 20-25k chars per contract) and asks, per document, for the decision, the enforcing mechanism, the admitted exception and the most consequential number, each quoted verbatim and prefixed with the file name, plus one cross-document dependency finding and a two-sentence summary (min_items:findings:12). The grounded check is ONE regex: alternation per contract over one distinctive token per document, drawn from the seat-extracted anchor corpus (Ecosystem/Benchmarks and Optimizations/ 2026-09-17-h04-adr-anchor-corpus/anchors.md, 195 anchors re-verified verbatim against the ADR bytes when the set was built): a figure with four or more digits or a separator (1568, 19.53, 0.307, 10,338), else a code identifier from the mechanism anchor (evict_costs, LOCAL_OFFLOAD_ORIGIN, gpulease.QueueHint), never a one- or two-digit integer. Same inline context, timeout_sec 900 (three documents on a 4 tok/s seat need it):

local-offload delegate --contract contracts/digest-adr-hard-8.json --route local --pin-reason measurement

(--pin-reason measurement is what keeps --route local a pin: without a reason, local and remote are hints that placement may override, which a measurement of one seat cannot have.)

Built by 2026-09-01-vllm-lmcache-results/stage3d-drivers/build_adr_hard_set.py in the results folder; regenerate there when an ADR changes, never edit the JSON by hand (the ground truth the judge reads is the inlined bytes).

digest-8-grounded.json — the same eight digests with a doc-only anchor each (register D-100)

digest-8.json's acceptance is SHAPE-ONLY (min_items:findings:3 + nonempty:summary; the intake lint says so on every run), so an 8/8 on it proves the loop completed and the schema filled, not that the digests are right. This fixture adds ONE regex: alternation per subtask over the document's CENTRAL identifiers — terms the document names and the goal does not (ADR 0025: llamaclient, modelaffinity, base URL, agent_model; ADR 0027: gpt-oss-120b, gpt-oss-20b, 1919, num-tokens, FlashML; and so on), every alternative checked present-in-document and absent-from-goal when the fixture is built. A digest that names none of a decision's key identifiers is shallow and fails verification; one that names any of them is grounded. Why an alternation, not one token (v2, 2026-09-15): v1 carried one contains: per subtask on a token that occurs exactly ONCE in its document (agent_model is a table cell of ADR 0025, gpt-oss-20b a passing comparison in ADR 0027). Measured on quiet seats the same day, correct digests skipped them: the 9B passed 2/8 and the 4B 4/8 while every contract completed and the spot-checked digests were right — v1 failed correct digests, which is the one thing an acceptance may not do. Run it BESIDE the old fixture (same seats, same day) for three K×8 passes before it replaces the old one: a step in pass rate is then attributable to the acceptance, not to the seats. Same inline context, no --read-root:

local-offload delegate --contract contracts/digest-8-grounded.json --route local --pin-reason measurement     # this box's seat
local-offload delegate --contract contracts/digest-8-grounded.json --route remote --pin-reason measurement --remote http://<node>:18811

Changing the inline text of a digest set

The three digest sets carry this repo's ADRs inline, so an ADR edit can leave a fixture's copy stale. A change that only substitutes names in the inline context text (and in the goal and context[].name of a subtask that names a renamed ADR file), with every acceptance array byte-identical (provable with a diff of the parsed JSON), is exempt from the side-by-side rule above: that rule exists so a step in pass rate is attributable to the acceptance, not to the seats, and a name-only substitution moves neither. Any other change, a rebuilt digest-adr-hard-8.json that selects different ADRs included, still takes three K×8 passes beside the old file.

Each set is sha-pinned by the sha256 of its committed (LF) bytes. A change records the old and new sha256 of every set it touches in the CHANGELOG entry of the release that carries it, and the old version stays reachable by commit id. TestContractFixturesAreWellFormed (root package) fails when a set loses a subtask, names a context doc that is not an ADR file, carries an acceptance string that does not parse or is UNGROUNDED, or has a regex: alternative that is missing from its docs or present in its goal. TestContractAcceptanceCarriesNoSpeedTerm reads every JSON file under this directory, the four templates, the four write-door contracts and the node list included, and fails on an acceptance value that is not a list of strings, a string the closed DSL does not parse, or a string with a speed term; it also checks that it read exactly the files that hold an acceptance key, so a walk that reads nothing cannot pass.