Say which stage-4 scripts are superseded and which are diagnostics

Seven scripts in this directory are not on the path to a number the paper
reports, but each is kept: four produced a published artifact, and deleting
them would leave that artifact in the tree with no code that made it, and three
are the diagnostics behind a conclusion the paper does state. Nothing said so,
so a reader had to work out which of two scripts answering the same question
was the current one.

Each now says it in its own docstring, and the directory README agrees:

  fewshot.py, fewshot_k0.py  superseded by fewshot_frac.py; FEWSHOT_CURVE.csv
                             came from them, and fewshot_frac.py mirrors their
                             subset construction including the row order
  live_pipeline.py           superseded by live_main.py, which is the source of
                             LIVE_MAIN_B.json; LIVE_PIPELINE.json came from this
  dataset_control.py         parts (a) and (b) are unique to it and current;
                             part (c) is superseded by ds_only.py
  rerun_score.py, cmp.py,    diagnostics, no published artifact; together they
  diag_live.py               traced the live-versus-matrix residual to vLLM
                             prefix caching rather than to routing

cmp.py and diag_live.py had no docstring at all and now carry one.

dataset_control.py's note records why its part (c) and ds_only.py's
`cheap + dataset` arm differ by about 0.0016 despite being the same nominal
configuration: the two build the corpus one-hot in a different column order,
and the routing heads are order-sensitive. Both numbers stay published, with
ds_only.py named as the one the paper reports.

No behaviour changes. Six tests pass and the manifest records the seven new
digests.

Author: zy06-learn

code/experiments/09_live_and_controls/code/README.md

@@ -17,11 +17,13 @@ scripts refit the main system from scratch, so that check is not a formality.
 | File | Question |
 |---|---|
 | `_contract.py` | The guard, and the two-level reproduction tolerance with the measured cross-platform deltas that justify it. |
-| `live_main.py`, `live_pipeline.py` | Is the reported test result read from a pre-computed matrix? 32,360 real verifier calls say no. |
-| `rerun_score.py`, `cmp.py`, `diag_live.py` | Where does the live-versus-matrix residual come from? It traces to vLLM prefix caching, not to routing. |
-| `dataset_control.py`, `ds_only.py` | Does the router receive dataset identity? Four arms, one of which hands it the corpus explicitly. |
+| `live_main.py` | Is the reported test result read from a pre-computed matrix? 32,360 real verifier calls say no. This is the source of `LIVE_MAIN_B.json`, the number the paper reports. |
+| `live_pipeline.py` | The single end-to-end pass of the same check, superseded by `live_main.py`. Kept because `LIVE_PIPELINE.json` came from it. |
+| `rerun_score.py`, `cmp.py`, `diag_live.py` | Diagnostics; no published artifact. Where does the live-versus-matrix residual come from? It traces to vLLM prefix caching, not to routing. |
+| `ds_only.py` | Does the router receive dataset identity? Four arms, one of which hands it the corpus explicitly. The version the paper reports. |
+| `dataset_control.py` | The same question in three parts. Parts (a) and (b) are unique to it; part (c) is superseded by `ds_only.py`, and the two differ by about 0.0016 because they build the corpus one-hot in a different column order. |
 | `fewshot_frac.py` | How much of its own pool does a held-out corpus need? Fractions, not counts. |
-| `fewshot.py`, `fewshot_k0.py` | The superseded absolute-count sweep, kept because the archived curve came from it. |
+| `fewshot.py`, `fewshot_k0.py` | Superseded by `fewshot_frac.py`. Kept because `FEWSHOT_CURVE.csv` came from them. |
 | `sig_main.py` | Paired cluster bootstrap for the main tables, Bonferroni over fifteen comparisons. |
 | `sig_controls.py` | The same test for the controls, with family sizes 1, 3 and 4. |
 

code/experiments/09_live_and_controls/code/cmp.py

@@ -1,3 +1,9 @@
+"""Per-verifier diff of a re-run against the frozen score matrix.
+
+DIAGNOSTIC. Writes no published artifact. Prints, for each verifier, max |delta|,
+whether the scores are exactly equal, their correlation, and old versus new latency.
+Used with rerun_score.py and diag_live.py to trace the live-versus-matrix residual.
+"""
 import os
 import pandas as pd, numpy as np, sys, glob
 # AFR_ROOT names the repository code root. The default is derived from this file's own

code/experiments/09_live_and_controls/code/dataset_control.py

@@ -6,7 +6,11 @@
     what the router could infer). Reported honestly: the features are document statistics, so a
     high number here is expected and is not leakage.
 (c) The control that matters: give the router the dataset identity explicitly as extra one-hot
-    features and re-run the whole pipeline. If AUROC does not improve, explicit dataset identity
+    features and re-run the whole pipeline. SUPERSEDED by ds_only.py, which runs this as one
+    of four arms and is the version the paper reports. The two build the corpus one-hot in a
+    different column order, and the routing heads are order-sensitive, so this file's arm (c)
+    and ds_only.py's `cheap + dataset` arm differ by about 0.0016. Parts (a) and (b) below
+    are unique to this file and are still current. If AUROC does not improve, explicit dataset identity
     buys nothing beyond what x already carries.
 """
 import os, sys, json

code/experiments/09_live_and_controls/code/diag_live.py

@@ -1,3 +1,9 @@
+"""Row-level diff of the live-selected scores against the stored matrix.
+
+DIAGNOSTIC. Writes no published artifact. For each action the router actually chose,
+reports max |delta| against the stored score and the first rows that disagree. This is
+where the vLLM prefix-caching residual is visible per row.
+"""
 import os
 import numpy as np, pandas as pd, sys
 from pathlib import Path

code/experiments/09_live_and_controls/code/fewshot.py

@@ -1,6 +1,11 @@
 #!/usr/bin/env python3
 """Few-shot adaptation curve, replacing leave-one-dataset-out.
 
+SUPERSEDED by fewshot_frac.py, which sweeps fractions of each corpus's own pool rather than
+absolute counts. The paper plots FEWSHOT_FRACTION_CURVE.csv. This script and its output
+FEWSHOT_CURVE.csv are kept because the archived absolute-count curve came from them, and
+fewshot_frac.py mirrors this file's subset construction exactly, including the row order.
+
 For each held-out corpus c the router is trained on the other three corpora in full plus k
 labelled rows drawn from c, and evaluated on c's own test split. k = 0 is the LODO case; the
 curve shows how many in-domain examples are needed before the held-out corpus stops being

code/experiments/09_live_and_controls/code/fewshot_k0.py

@@ -1,5 +1,9 @@
 #!/usr/bin/env python3
-"""The k=0 point the main sweep skipped: pure leave-one-dataset-out."""
+"""The k=0 point the main sweep skipped: pure leave-one-dataset-out.
+
+SUPERSEDED with fewshot.py by fewshot_frac.py, whose fraction grid starts at 0.00 and so
+carries this point already. Kept because FEWSHOT_CURVE.csv came from the pair.
+"""
 import os, sys, json
 from pathlib import Path
 import numpy as np, pandas as pd

code/experiments/09_live_and_controls/code/live_pipeline.py

@@ -1,6 +1,11 @@
 #!/usr/bin/env python3
 """End-to-end LIVE run of the deployed pipeline: no pre-computed test matrix.
 
+SUPERSEDED by live_main.py, which runs the same check per (row, seed) over all ten seeds and
+is the source of LIVE_MAIN_B.json, the number the paper reports. This script runs the single
+end-to-end pass and writes LIVE_PIPELINE.json; both are kept so each published artifact has
+the code that produced it.
+
 Training uses the frozen TRAIN scores (that is what training is for). At test time the script
 computes the routing decision from cheap features alone and then calls EXACTLY ONE verifier per
 instance, live. The other two verifiers are never invoked for that row, and their columns in the

code/experiments/09_live_and_controls/code/rerun_score.py

@@ -1,6 +1,10 @@
 #!/usr/bin/env python3
 """Independent re-run of the three pool verifiers on a sample of the frozen scoring cohort.
 
+DIAGNOSTIC. Writes no published artifact. With cmp.py and diag_live.py this is what traced
+the live-versus-matrix residual to vLLM prefix caching rather than to routing, which is the
+conclusion the paper reports; the scripts are kept so that conclusion can be re-derived.
+
 Writes to a fresh output_dir so the harness cache cannot serve stale rows: this is a genuine
 re-execution, not a resume. Nothing under ingest_and_scoring/results is touched.
 """

code/experiments/MANIFEST.sha256

@@ -198,16 +198,16 @@ aeb4d37a8f82ad8a7a4991e2bf753e8b8de3ef6bfa8d4194cce069b287a0521b  ./08_routing_c
 7a53e7c0dfadecbc5b32276149b0beeb5801d88540c0ec3e4105c5c9b25e6179  ./08_routing_code/v3core.py
 bb1a506e00cc321a849ccc5008c6a765e86d4bb13aba03ff8913447003d2aa06  ./09_live_and_controls/README.md
 0c450a535bbf67901ca47d6fce4221314c860981b665539eab202f6d71ce36c3  ./09_live_and_controls/code/_contract.py
-cc476ddceb684049c39b524d4417bd2f2cfe7cc6dcafc95f56fbacacc2b00849  ./09_live_and_controls/code/cmp.py
-4f9629600eec5ca85cea48e88d50db181023246321e00c8ef40cdf766d8ead8d  ./09_live_and_controls/code/dataset_control.py
-1897cbb5b775e70e8d13b2dcb9a33eb156a023fa18e9956f520e64b51920346c  ./09_live_and_controls/code/diag_live.py
+30ba2b9e7a5b095feabd1e946229cc19abedb52e9b33206c053ca1d193a5f10b  ./09_live_and_controls/code/cmp.py
+5edbd37298e567be9a1d140c37a7d551dbd32a7c3faf1e26adfe070ab34aba02  ./09_live_and_controls/code/dataset_control.py
+8471bcd424a07f6086c3efcb8917a975b834843a6477970fe306de07a7c42f39  ./09_live_and_controls/code/diag_live.py
 e5beefa3b40f957c3024f6b1a644ed52ba92cafc680eec325b8ea13a118d51a8  ./09_live_and_controls/code/ds_only.py
-a5efec4d504f7c632c8f0876b44efe011708c2c460af6d719e600153bb20f479  ./09_live_and_controls/code/fewshot.py
+948f341aa03d15b362709cda06781166a802199a44fbc64b876c316c305fa8b1  ./09_live_and_controls/code/fewshot.py
 2024106b6963f18c268864b87205bfbecc392ed29898679c745c405fe929a021  ./09_live_and_controls/code/fewshot_frac.py
-b54d42e127ddc489ca425d4c39ed3dde6fa8cfc5c2127c93ca92f3dd6e1c663c  ./09_live_and_controls/code/fewshot_k0.py
+66e30899f00fed0b2ffccd8d8853b4805a41092f6bfe3c7fa73c6e1909188d84  ./09_live_and_controls/code/fewshot_k0.py
 adca9229f9494a83799ff058639e4a2540a15b07defd20552da5718baffa78c4  ./09_live_and_controls/code/live_main.py
-aa1f737a6d6cee7086effbb89915d9dd90ed18722a2f250db0796d1fcec2fa1f  ./09_live_and_controls/code/live_pipeline.py
-49fc5ff3a863c894b47b585c66e7fc0edd6f581981c9f374dc295503e62375b1  ./09_live_and_controls/code/rerun_score.py
+fdf8280d82b43e4c56f1808933005c619c6349c875ec36a1cf27e02a01d911e6  ./09_live_and_controls/code/live_pipeline.py
+b00d798dacb710db00e70b2d3ce27a1ebb9301218cf26ef9ba401facc836116a  ./09_live_and_controls/code/rerun_score.py
 578955518c4c35ee40a9fe3194ed2ecfa5f7954a46bf18a36c577b999b631676  ./09_live_and_controls/code/sig_controls.py
 5e9603ef4585fcc45bc33b9db6cedae60a94e0595055a14868393118a62626ce  ./09_live_and_controls/code/sig_main.py
 34a1efdb08b66f1ca14ce250ef6567355f37a6933c862e422f68274b66659e58  ./09_live_and_controls/results/A_SIGNIFICANCE.csv