Skip to content

tasks.md

Latest commit

 

History

History
190 lines (158 loc) · 13.2 KB

tasks.md

File metadata and controls

190 lines (158 loc) · 13.2 KB
Raw

Frozen walker instruments

Six instrument shapes (Shapes 1–4 as first published on 2026-08-03; Shape 5, Shape 6 and the added Shape 2 variants below Shape 4, added 2026-10-22). Each is reproducible against the named blob; none is pooled with another.

Shape 1—lane EXECUTION (Run 11's EN frame)

Walks how a lane's rails execute (gates, prohibitions, fail-routes, writes) from a frozen contract prompt. Common wrapper, byte-identical across every dispatch; only [SCENARIO BLOCK] differs.

Contract walk — READ THE SKILL AND REPORT WHAT YOU WOULD DO, IN WHAT ORDER. DO NOT ACTUALLY DO ANY OF IT.

## What to read (read all of it before answering)
<path to the SKILL.md under test>
If it points at another file and you judge that file necessary to decide the situation below, you MAY read it — then say which file you read and why.

## The situation to walk
[SCENARIO BLOCK]

## What to answer — answer as numbered items EXACTLY as listed. Do not add headings. Do not skip an item.
1. Does this skill's activation condition apply? — yes/no, and which sentence in SKILL.md made you decide that (quote it).
2. The sequence of steps you would take — a numbered list to the end; walk ONE path; if information is insufficient, follow the skill's own unknown/default rule and say where you got stuck.
3. Points where you must ask the user's permission first — how many, where, what each asks.
4. Points that are absolutely forbidden — list every prohibition SKILL.md states.
5. Where the output goes — which file, which folder.
6. If something goes wrong mid-way — what does the skill tell you to do.
7. Any grade / level / mode you must choose — if so, what did you choose and why.
8. What in SKILL.md you read and were NOT sure about — quote the ambiguous sentence, or write "none".

## RAILS
- Never actually do the work — no spawn, no write, no edit, no state-changing command. Read-and-report only.
- Do not search the web. Do not read any other repo.
- Answer from what you read in the file — if SKILL.md does not say it, write "SKILL.md does not say".
- Answer in English, except technical terms and text quoted from the file.

Full scenario blocks (4 lanes: routing-OFF, sensitive/never-down, escalate-up, route-to-self) and the per-run results live in results/2026-08-03-skill-determinism-walks.md §Run 11 (this benchmark's detail file consolidates the prior lab record—the canonical working copy is TheColliery/scratchpad/longrun/SKILL-VARIANCE-WALK.md, machine-local and not part of this public repo).

Shape 2—lane SELECTION (Run 12's shape)

Walks the decision UPSTREAM of execution—given a situation, which entry does the reader pick at all. 4 scenarios (one real, three controls), EN, ~1.1k chars, questions A (routing pick) / B (enumerate the trigger list) / C (free-text criticism). Not pinned at fire time (declared gap in the original record)—reconstructable from the scenario summaries in the detail file, not byte-reproducible the way Runs 13-15 are.

Shape 3—hook-injected coordination cue (Runs 13, 14, 17, 18)

Walks a UserPromptSubmit-injected cue, not a SKILL.md body—the load layer is two hook-emitted lines. 5 scenarios, EN, answer format A (which leads) / B (next action) / C (surface to user Y/N) / D (enumerate the stakes set) / E (self-reported uncertainty). Same 5 situations and answer format reused across all four waves—the clause text is the only thing edited between runs, which is what makes the runs like-for-like; it also means the situation set has now been iterated against three times by the same author (see RESULTS.md §Publishability verdict for what that bounds).

  • Run 13 (before carve): results/raw/run13-prompt-v1.txt, blob 3b4ff7080d7ccf935a32aa254bcc1c3bdd32b91c, 3,760 B.
  • Run 14 (after wave-1 carve, hook line A substituted, scenarios untouched): results/raw/run14-prompt-v1.txt, blob cd7c82405ffa9ad4c40ac9ade85ad471ce904d52, 3,999 B.
  • Run 17 (after wave-2 carve, both hook lines substituted—one-flock means CoalTipple's copy now carries the same clause too, so this is NOT like-for-like with Run 14 on that one axis): results/raw/run17-prompt-v1.txt, blob e82b0693ec2ea28998ecd8b93da6522474ee8cc4, 4,322 B. Clause walked: 594 ch, sha 23fed45403bb, byte-identical across both conductors and both plugin/ dists at fire time.
  • Run 18 (after wave-3 carve, both hook lines substituted, generated by reading the clause out of the live shipped conductor rather than hand-pasted): results/raw/run18-prompt-v1.txt, blob 70a3ecd9a33efb5f0d9b8ac8c0b855d74089b855, 4,408 B. Clause walked: 637 ch, sha 2416c04309ce, byte-identical across both conductors and both plugin/ dists at fire time.
  • Run 19 (wave-4 clause, generated by substituting the live shipped clause into both hook lines, three fail-loud guards asserted wave-4 present / wave-3 absent before any walker spawned): results/raw/run19-prompt-v1.txt, blob 93e42c03d5b2d60e2390c78a1411350d9aad6472, 4,504 B. Clause walked: 685 ch, sha 4a18bb6f8f03, byte-identical across both conductors and both plugin/ dists at fire time.
  • Run 20 (wave-5 clause, two additive phrases on top of wave 4, four fail-loud guards): results/raw/run20-prompt-v1.txt, blob 4da42a7f2275337b3b3db8ca993ac8e4373328d5, 4,842 B. Clause walked: 854 ch, sha1 e72fec15dffe, byte-identical across both conductors and both plugin/ dists at fire time. Reverted the same day (CoalBoard 2b99b46, CoalTipple 85f1312) back to the wave-4 clause—this prompt is retained as the frozen record of what was walked, not of what ships.

Shape 4—this benchmark's prose-index hypothesis test (Run 15)

Same shell as Shape 1 (8 numbered questions, read-only rails), scenario constructed so every branch of the target skill's own deterministic tier rubric is satisfied by the prompt's own wording—a clean test of whether the walker COMPUTES the branch rather than guessing it.

  • Run 15 (CoalMine rot-canary, prediction locked before firing): results/raw/run15-prompt-v1.txt, blob c9ccd10f9f644796d77f853b5006e58f95696437, 2,874 B.
  • Target walked: installed coalmine@coalmine 3.14.0 skills/rot-canary/SKILL.md, blob 57406d5a42f55474672bbbed3d31a62a86cadd59.

Instruments added 2026-10-22 (Runs 21–46 and two later walks)

Per-run blobs, keys and tier compositions: results/2026-10-22-skill-determinism-walks-21-46.md §Master stamp. Every published prompt below is a copy of the dispatched file with one line replaced: the line naming the local path of the file under test became FILE: <path to the SKILL.md under test> (Run 12's template has no such line and is byte-identical). Anyone can reproduce the dispatched bytes by putting a path back on that line and checking the dispatched blob.

Shape 2, template run12-v2 (lane SELECTION; Runs 21, 22, 44, 45, 46): four scenarios answered with one token each, an offer-vs-convene answer, an enumeration of the auto-trigger conditions, a self-report. Runs 21 and 22 substituted the skill body into the prompt ({SKILL_BODY}); Runs 44–46 had the walker Read the file by path (a declared deviation). The template is a reconstruction of the Run 12 instrument, which was never frozen.

Shape 2, disposition v1 (Runs 25, 26, 28): four scenarios, each answered ACT or ESCALATE after a board verdict, plus a count of the escalation conditions. Prompt text not retained as a file; the record holds its hash (sha256/12 3830ec8365c4, 2,321 B) and the scenario descriptions in the detail record. Not byte-reproducible.

Shape 5, enumerated-rail count (Runs 24, 27, 29, 30, 31, 34–43): the walker reads one SKILL.md, answers for a stated scenario, and gives a count and a list for each of: consent gates, prohibitions, output locations, fail/degrade paths, config keys. Dispatch rules that changed during the series: a bare COUNT: line (Run 35), then numbered COUNT-Q<n>: markers and a lane-scoped question (Run 39 onward); a supplied definition of a consent gate (Runs 36 arm 2, 39–43); load layer pinned to the body (Run 35 onward). Prompts for Runs 24 and 27 are quoted in the detail record (Run 27's is abridged there); Runs 29 and 30 reuse Run 27's; Run 31's was not retained (scenario described in the detail record).

Shape 6, categorical rail reading (OW-1, OW-2): the walker reads the edited rows and returns one JSON object answering named questions; the key is a table of expected answers written before any walker runs.

Published prompt files

File (in results/raw/) Used by Dispatched blob (bytes) Published blob (bytes)
run12-prompt-v2.txt Runs 21, 22, 44, 45, 46 7114936a620d (2,369) same, byte-identical
run34-rotcanary-prompt-v1.txt Run 34 d5aef97b2bbb (2,058) 0b1cd2ac5813 (2,009)
run35-agent-prompt-v1.txt Run 35, agent lane 269be0363251 (1,471) da452b1ec520 (1,422)
run35-hook-prompt-v1.txt Run 35, hook lane 07f66f26797c (1,540) 3cc50490a4c2 (1,491)
run36-gs-agent-prompt-v1.txt Run 36 arm 1; Run 38, agent lane 673f68db8fbb (2,022) 86439bb06a0d (1,970)
run36-gs-agent-defined-prompt-v1.txt Run 36 arm 2 c6095d64d353 (1,348) 08518db018d2 (1,296)
run36-gs-hook-prompt-v1.txt Runs 37, 38, hook lane 36b4284b60df (2,119) 91b1254fde6d (2,067)
run39-sg-agent-prompt-v1.txt Runs 39, 40, 41, 42, agent lane 941049a622fe (2,351) e21d4bf5130e (2,296)
run39-sg-hook-prompt-v1.txt Runs 39, 40, 41, 42, hook lane 0c51cf02a197 (2,383) a254aae34a30 (2,328)
run43-sg-q3q4-agent-v1.txt Run 43, agent lane 286859b96a72 (1,493) 48e4ac9f5015 (1,438)
run43-sg-q3q4-hook-v1.txt Run 43, hook lane 386cd3ca88d7 (1,525) 135b60e7087e (1,470)
ow1-coalwash-consent-ledger-prompt-v1.txt OW-1 0d30a6f4d95d (1,541) ac576c978d5d (1,491)
ow2-coalwash-keeps-row-prompt-v1.txt OW-2 2a17146a7b5a (1,523) b3ecb6dd6949 (1,464)

Blob = git blob id of the file's bytes (git hash-object <file>), 12 hex characters. The target blobs the prompts were run against are in the detail record's master stamp; where a stamp is a plain sha1sum of a CRLF checkout (Runs 44, 46), the detail record gives the re-derivation command.

Reproducing a run

  1. Fetch the target blob at the pinned hash (the installed plugin cache, or git cat-file blob <hash> against the tool repo's history if the blob is a commit-reachable object there).
  2. Dispatch the frozen prompt to N fresh, memory-less, no-spawn leaves per tier (this record used blind-ic under Claude Code; any zero-context, zero-tool-beyond-read leaf qualifies).
  3. Score each rail's modal answer and membership per the formula in PREREGISTRATION.md.
  4. Walker transcripts are collected into the detail record, then destroyed (standing lab-hygiene rule)—only the frozen PROMPT and the scored TABLE persist; a stranger re-running this gets a fresh, independent reading rather than an answer key.

Scorer note (fifth instrument defect, Run 20, fixed)

A line-anchored scorer (matching on an S<n> label) silently drops any walker whose answer uses a different shape and reports the aggregate over the survivors—this happened five times across the series (Runs 17, 18, 20 caught; two more shapes seen in raw output but not mis-scored). Run 20's instance was the most dangerous: it dropped exactly the two weak-tier walkers who could not follow the text, which manufactures an apparent improvement on the rail that measures whether the weak tier follows the text at all—the direction opposite of a random drop. The replacement scorer is format-agnostic (order-of-appearance, not line anchors) and refuses to print an aggregate unless all N walkers yield exactly the expected number of answers per rail — a parse failure fails loud instead of silently shrinking the denominator. Use the replacement for any re-run; do not revert to line-anchoring for convenience.

Scorer rules added 2026-10-22 (from Runs 34, 36 and 39)

  • The extractor is part of the instrument. Before any number reaches a table, read at least one answer per tier by eye. Run 34's first pass read 6 of 9 counts as unparsed; Run 36's read each count from the wrong question, because walkers put the COUNT: line before the heading or emitted no headings.
  • Extract by ordinal, mark by number. The Nth COUNT in document order is question N; from Run 39 the prompt requires COUNT-Q<n>: markers so position is never inferred. 18 of 18 answers parsed on the first pass in each of Runs 39–43.
  • Score count and membership. A rail's count can be identical across a tier while the sets differ (Run 34, weak tier 2 / 2 / 2).
  • Report the divergence and the modal answer for every cell. They are two measurements (Run 41).
  • Write the key before the first walker; amend it only because the walked text changed, and record the amendment before the next walker runs.
  • A prompt, its scorer and its key change between skills, never between one skill's before and after.