Ground the flywheel studies in real recorded results#2
Open
TaylorAmarelTech wants to merge 68 commits into
Open
Ground the flywheel studies in real recorded results#2TaylorAmarelTech wants to merge 68 commits into
TaylorAmarelTech wants to merge 68 commits into
Conversation
…en judge audits Adds the post-merge Codex flywheel surface: Gemma 4 adapter study collection and notebook builders (six locally executed CPU notebooks), TPU LoRA training lab and grounded-scratch TPU notebook builders, measured-response curriculum builders, four-arm evaluation with recorded egregious-example join, frozen frontier-judge and recorded harmful-request judge studies, the DueCare system evidence receipt, and a governed model-fallback registry with validator and public-surface check. Study summaries now report exact two-sided sign tests and evidence-scale labels beside bootstrap intervals so anecdote-scale pair counts cannot be over-read; notebooks compute the same statistics from recorded verdict counts, add pair-level criterion consistency (adapter vs harness kept separate), and keep JSON artifacts strict (no NaN). The multiperspective bundle builder gains the four-way curriculum-focus expansion, lineage fields, minimal near-miss preference negatives, and a content-similarity gate. llm_generate omits non-default sampling parameters for Anthropic routes. The Kaggle publish and training-run payload directories are now gitignored.
…xonomy analyze_full_results now joins the accumulated judged panel to the real 78,719-prompt registry: per-model statistical strength (seeded bootstrap 95% interval, exact/normal-approximation two-sided sign test, Wilson win-rate interval), per-judge robustness (the paired lift recomputed independently inside each judge's verdicts), and lift breakdowns by real prompt category, corridor, and difficulty, with the complete aggregate published as a JSON sibling so capped tables drop nothing. Current read: gemma4:31b baseline 48.4 -> harness_core 89.1 (+40.7 [+40.2, +41.2]) over 7,953 paired real registry prompts, consistent within all three judges, largest lift on the hardest prompts, and full - core = -1.04 at n=7,958. build_recorded_harmful_request_expansion freezes every qualifying recorded Gemma failure (46 of 52 ranked prompts have complete recorded pairs) from the actual egregiousness ranker and rich-lift ledger into a judge-ready pack with 92 blinded two-order requests pinned by canonical hash. The pack is prepared_not_executed, local evidence only, and training-ineligible; the recorded-study verifier now accepts expansion manifests, and the four-arm join gains an allow_partial mode that still refuses empty packs.
Fixes the four-vs-five public dataset count and the split showcase table in the Kaggle index; states in the README that the frozen 32-verdict judge audit did not support a positive training-lift claim so the +9.67 harness result cannot be misread as a fine-tune win; adds the live-panel read (7,953 paired real registry prompts, +40.7 with interval and per-judge robustness) beside the checksummed receipt numbers; documents the exact-sign-test upgrade and the prepared 46-pair recorded harmful-request expansion pack, including its prepared_not_executed and local-only boundaries.
…re signaling Addresses defects from a multi-agent survey of the committed flywheel: - training_contract: the unsafe-advice gate cleared an operational-uplift phrase if any negation appeared anywhere in the answer, so a far-away refusal could launder an operational instruction. Negation must now sit within a bounded window of the operational match (_operational_advice_is_unmitigated). Scans the chosen/assistant answer only, never the preference rejected side. Strictly fail-closed; all release/bundle suites still pass. - A-00 kernel: the canonical release verifier is exec'd from a repointable source root with no hash pin. It now records the verifier module SHA-256 in the verification receipt (canonical_verifier_sha256) and fails closed when DUECARE_A00_EXPECTED_VERIFIER_SHA256 is pinned and does not match. - ollama_adversarial_flywheel: a whole-run failure (e.g. Ollama down) previously wrote a 'successful' manifest with zero candidates. The manifest now carries run_status + exception_failures, and the CLI exits non-zero on failed_no_candidates. - build_gemma4_adapter_kaggle_collection: the default run list named future regime runs that resolve(strict=True) turned into a hard crash. resolve_run_ directories now skips missing runs (recording their names) while still enforcing the two-completed-run minimum; the regime matrix is optional. First test coverage for this packager. - Removed a rule-60 display truncation (bounded[:480]) in the showcase notebook.
The docs/README/Kaggle-index assert a release-manifest SHA-256 and exact row counts per published dataset that were otherwise unverifiable prose. Adds: - configs/duecare/training/published_dataset_claims.json: committed single source of truth (5 datasets, each SHA + row counts + staged-artifact globs). - scripts/verify_training_dataset_claims.py: re-derives each SHA-256 from the gitignored staged/re-downloaded release manifests and re-checks row counts, failing closed on mismatch; a clean checkout reports each as published-only. All 5 claims verify against local artifacts. - validate_public_surface.py: a published_dataset_claims check that fails if a registry SHA drifts out of the training doc (clean-checkout safe; now 9 checks, 0 findings). - docs/training_and_finetuning.md: points at the registry + verifier so the (git_sha, dataset_version) reproducibility invariant is executable.
Owner
Author
Second pass — hardening from a 4-agent survey of the committed flywheelPushed two more commits ( Correctness / safety gates
Reproducibility / credibility
Verified, not changed: the safety/harness headlines (+9.67 recorded, +1.73/+0.18/+4.39 receipt) are over the real registry/ledger/gallery; only the adapter +0.15 is synthetic mechanism-scale (honestly labeled). No script pushes to Kaggle. The Gates: 92 changed-area tests pass · packages collect 1,306 · public surface 9 checks/0 findings · ruff clean on all new/edited files. |
Extends the training-dataset claims verifier per two review forward-pointers: - The adapter-study registry entry now pins the seven headline numbers that live as hardcoded prose in the study notebooks (+1.73 model judge, +0.18 deterministic, +4.39 adversarial, +9.67/6-of-6 recorded, 60 steps, 817,152 trainable params, +0.15 objective). verify_training_dataset_claims.py re-derives each from the staged system-evidence receipt / run metrics and fails closed on drift, so a re-run that changes a number can no longer leave the notebook prose silently stale. All 7 re-derive locally. - Adds an opt-in, credentials-gated --check-live flag that does a read-only 'kaggle datasets metadata' lookup to confirm each id resolves to a live public dataset. Off by default and never run unprompted; it makes no publish call. This closes the residual 'offline can only prove self-consistency, not live match' gap without any automatic Kaggle traffic.
Implements the central recommendation of the 2026-07 harness/finetuning blueprint: turn DueCare's hard-coded intelligence into a governed, auditable supervision fabric that is never mistaken for ground truth. - duecare.chat.rulecards: compiles each raw GREP indicator rule into a typed RuleCard (authoritative sources, antecedent, consequence, inferred role, jurisdiction, witness family, calibration gaps). Every rule classifies as a fallible labeling_function + feature_extractor; none are auto-promoted to a hard invariant (a match is grounds for inquiry, not proof). Critical rules are flagged for human invariant review, never silently. - independence_report measures the blueprint's load-bearing insight from the REAL deck: 451 rules resolve to 80 correlated-witness families; the largest holds 86 rules and the top five hold 63% of the deck (Palermo 153, ILO C181 113, C029 89). Same-family rules are correlated votes, not independent confirmations — the input a correlation-aware weak-supervision label model needs so one legal principle expressed as many patterns cannot inflate confidence. - scripts/build_rulecards.py compiles the live rules + their CATEGORY headers into a deck, summary, independence JSON, and a markdown report (deterministic, no model call; output gitignored). - docs/research/rulecard_supervision_fabric.md reconciles the full blueprint against DueCare's measured state (451 rules, +40.7 real-panel harness lift, the honest fine-tune result) with a build order grounded in what exists. 9 module tests; packages collect 1,315; public surface 9 checks/0 findings.
The showcase notebook quotes +1.73/+0.18/+4.39/+9.67 as narrative prose while its charts read the same numbers from JSON. The load cell now re-derives all four from the packaged system-evidence and recorded-safety receipts and raises if any drifts, so a re-run that changes an artifact can never leave a stale hardcoded figure in the narrative. Executes clean (0 errors) against the real receipts.
Owner
Author
Third pass — RuleCard supervision fabric + blueprint reconciliationBuilding on the harness/finetuning deep-research blueprint, with everything grounded in the real repository (per the standing "avoid too-synthetic" directive). RuleCard supervision fabric (the blueprint's central recommendation)
Claim provenance (both reviewers' forward-pointers)
Gates: 78 changed-area tests pass · packages collect 1,315 · public surface 9 checks/0 findings · ruff clean · study notebooks re-execute with 0 errors. Note: |
Extends the RuleCard fabric with the blueprint's correlation-aware effective witness count and packages it for publication: - rulecards.effective_witness_count applies the design-effect formula m/(1+(m-1)*rho) per witness family and sums across families, so the structural 80-family count (rho=1 bound) is shown alongside softer correlation assumptions. On the real deck: ~83 effective witnesses at rho=0.9, ~90 at 0.7, ~100 at 0.5 -- far below 451 raw rules. independence_report now carries effective_witnesses_by_rho with an honest bounds note. - scripts/build_rulecards_kaggle.py packages the deck as a public CC-BY-4.0 Kaggle dataset (typed deck, independence report, summary, flat CSV, data card, license, SHA-256 release manifest) plus a CPU visual notebook that renders the reduction result, effective-witness curve, per-instrument concentration, severity/review boundary, and category coverage. Rule metadata only -- no worker data or PII. Locally executed clean (0 errors, 4 charts).
The RuleCard supervision-fabric dataset is now live and public at taylorsamarel/duecare-rulecard-supervision-fabric (CC-BY-4.0, 451 RuleCards, 80 witness families, effective-witness range). Sets isPrivate:false on the dataset metadata and gives the companion visual notebook a distinct id/title (duecare-rulecard-witness-atlas) so it no longer collides with the dataset slug.
Owner
Author
Fourth pass — effective-witness math + first Kaggle publishBuilding on the full deep-research blueprint, with Taylor's explicit authorization to publish to Kaggle this session. Effective independent-witness count (blueprint §2.2) Published — live and public on Kaggle
Gates: 11 RuleCard tests pass · ruff clean · public surface 9 checks/0 findings · dataset PII scan clean (rule-name matches like Six commits this session's later waves; branch pushed. The RuleCard dataset is the first genuinely novel, honest, public artifact from the supervision-fabric direction. |
…e notebooks Turns the thousands of real Ollama/multi-source gradings into a public research asset and reproducible example notebooks -- maximally using the graded panel. - build_benchmark_results_kaggle.py packages the real 85,417-row graded panel (0-100 rubric score + A-E component scores per model/arm/prompt/judge) as a CC-BY-4.0 Kaggle dataset with SCORES ONLY -- no response text, no prompt text, no PII -- plus joinable prompt metadata (category/corridor/difficulty labels) and the precomputed harness-lift analysis. Rich metadata for usability: subtitle, 200+ char description, keywords, and per-column schema descriptions. - Two runnable example notebooks (attach the dataset, no internet): 'Reproduce the Harness-Lift Result' recomputes the +40.7 lift from scratch with bootstrap CI, exact sign test, Wilson win-rate interval, per-judge robustness, and category breakdowns; 'LLM Judge Agreement' measures pairwise judge correlation (0.905-0.925 on the real grades) and per-component score patterns. Both locally executed clean (0 errors, 5 charts). - 3 tests: prompt-text-never-leaks safety, usability metadata + column schema, and runnable notebook contract.
…on, column schema) Both DueCare datasets now carry full usability metadata: a <=80-char subtitle, a 1000+ char description, four valid-vocabulary keywords, per-file descriptions, and per-column schema descriptions for every CSV. Kaggle scores these directly. Constraints learned and encoded: keywords <=6 (used 4 valid slugs), subtitle 20-80 chars. Both datasets republished public.
Kaggle derives a kernel slug from its title; a title that does not slugify to
the metadata id fails the push ('title does not resolve to the specified id').
The example-notebook titles are now bare 'DueCare Reproduce Harness Lift' /
'DueCare Judge Agreement' so they slugify to exactly their ids.
…s read analyze_full_results now cites the public grades dataset + reproduction notebooks in the generated report header (the report is regenerated, so the link belongs in the generator, not the .md). Also adopts the datetime.UTC alias via linter autofix.
…d deck Dual-use safeguard. The public rulecard-deck.json exposed the exact regex patterns for all 451 rules -- an operational detection map that could help offenders rephrase to evade it. The project's safety rules and the finetuning blueprint both forbid publishing evasion-enabling operational detail. The public deck now carries pattern_count (not the raw regex) per card while keeping all research metadata (legal sources, severity, roles, jurisdictions, witness families, indicator reasoning); the redaction is disclosed in the README. The full-pattern deck remains available internally for DueCare's own use. 3 tests guard against regex leakage.
Makes DueCare launchable in one step for NGOs, developers, researchers, and notebook users, and fixes a real containerized-launch bug found while reviewing the deployment surfaces: - FIX: the Docker ENTRYPOINT was 'python -m duecare.chat.run_server', but run_server was only a FUNCTION -- no such module existed, so the container crashed on start. Adds the missing runnable module packages/duecare-llm-chat/src/duecare/chat/run_server.py (a thin argparse CLI around the existing run_server()). 'python -m duecare.chat.run_server' now works; the image starts. - launch.py: one friendly launcher mapping audience -> command (workbench / ngo / demo / benchmark / notebook), with prereq detection and --dry-run. 6 tests. - docker-compose.build.yml: a build-from-source override so a fresh clone can 'docker compose up --build' without needing the published GHCR image (verified valid with docker compose config). - docs/QUICK_LAUNCH.md + a notebook launcher (examples/deployment/notebook/) that runs the server in Colab/Kaggle/Jupyter and exposes a public tunnel URL. - README quick-start now points at the one-command launcher.
Implements the confirmed methodology findings from a PhD-stats review: - Leave-one-judge-out lift envelope (the judge-averaged bootstrap marginalizes out judge variance, so [+40.2,+41.2] was false precision). The envelope [+40.4,+40.9] is now the primary uncertainty statement; the prompt-only bootstrap is explicitly flagged as judge-blind. - Normalized gain (harness-baseline)/(100-baseline) controls for the rubric ceiling that mechanically compresses lift at high baselines (0.74 of headroom). - The sign test no longer renders 'p = 0.0' -- it shows '<1e-300' and is labeled direction-only; the win-rate Wilson interval carries a design-effect caveat (the same 3 judges recur across prompts). - 'serve core, not full' is now gated on a paired bootstrap CI that excludes zero; a ~1-point mean within noise is called statistically indistinguishable, never an operating rule. - The rubric-preamble/judge-rubric congruence confound (part of the raw +40.7 is instruction-following, not domain value) is surfaced in the report body with the length-matched placebo named as the next full-scale measurement. 5 new/updated tests. Direction remains well-supported and judge-robust; the magnitude and its uncertainty are now defensible as stated.
…reement Implements the confirmed prof-stats / eng-industry persona findings against the real 85,417-row grades panel (verified by --execute-local): - Judge-agreement notebook now reports WITHIN-ARM agreement (ICC(2,1) per arm + Fisher-averaged within-arm Pearson r) instead of pooled Pearson. Pooling across arms inflated r to 0.918; the honest within-arm r is 0.723, and ICC reveals judges agree strongly on baseline (0.80) but only moderately on the richer harnessed responses (0.42-0.45). The closing note no longer overclaims "high agreement" and points to the near-unanimous lift direction instead. - Both notebooks use a single canonical headline_model() (prefer gemma4:31b, else most-paired) so the lift and agreement cells never disagree on the head. - find_dataset() verifies the release-manifest dataset_id (binds to the right dataset even if another DueCare dataset is attached). - Sign test / Wilson interval guard the all-tied (info==0) and empty (n==0) cases; sign p renders as <1e-300 rather than a literal 0.0. - Lift cell raises a clear error instead of an IndexError when no model yet has >=5 paired prompts. Both notebooks execute end-to-end against the real panel: headline reproduces at gemma4:31b 48.4->89.1 (+40.7), sign test <1e-300 over 7,938 wins / 15 losses.
Applies the confirmed eng-industry findings to the public RuleCards supervision-fabric notebook, verified by --execute-local (0 cell errors): - find_dataset() now verifies release-manifest.json dataset_id against the expected slug, so the notebook binds to the right dataset even when another DueCare dataset is also attached (was: first subtree with the sentinel file). - Category chart filters 'uncategorized' BEFORE taking the top 16, instead of slicing first and then dropping it -- the old order silently lost a real category whenever 'uncategorized' fell inside the top 16. Title now reflects the actual named-category count. Rebuilt + executed against the real 451-rule / 80-family deck; effective witnesses 82.7-100.1 across rho, no cell errors.
… claims Domain/legal-expert review of the vetted claim library (85 claims) surfaced two overstatements that the freshness validator flagged: - sa_kafala_reform_2025: an advocacy-sourced (Walk Free), pending-human- verification announcement was tagged binding_domestic. Reframed the text to attribute the "end of kafala" to reporting and foreground that it is a REPORTED, partial reform (sponsorship persists in modified form, domestic workers largely excluded, prior Gulf promises outran implementation), and downgraded binding_status -> reported_reform_unverified. - eu_forced_labour_regulation: binding_status binding_domestic could let a consumer that reads only the status (not the caveats) present a rule whose market-ban obligations do not apply until 14 Dec 2027 as currently enforceable. Encoded it as enacted_not_yet_applicable. Both claims remain flagged for recheck (they still need primary-source human verification); this only makes any grounded use before then honest. Library still validates (85 claims, 13 tests pass); register regenerated.
…notebooks Kaggle's SaveKernel has a per-account DAILY push cap (resets 00:00 UTC); once hit, every `kernels push` returns 429 until the reset. This script makes publishing the corrected notebooks (judge-agreement, reproduce-harness-lift, rulecard-witness-atlas) re-runnable and scheduler-friendly: - stages each at a SHORT path + pre-creates the %TEMP% upload mirror (Kaggle CLI 2.2.x temp-path bug workaround), - distinguishes 429 (exit 3, "try after reset") from a genuine error (exit 2), - --wait-for-reset sleeps until just past 00:00 UTC then pushes, - --rebuild regenerates from the live panel/deck; --dry-run stages only, - verifies each slug against the account's live kernel list. Dry-run confirmed it finds + stages all three built notebooks.
The full-registry per-dimension sweep is a ~24-day-per-model job, so we want statistically valid INTERIM readouts long before it finishes. The promptset is built in a structured order (curated/stable head, then a stratified expansion), so a file-order prefix is a biased slice. This adds an opt-in seeded shuffle so any interim prefix of graded prompts is an unbiased random sample of the full registry -- interim goals reduce the prompt COUNT, never the grading resolution (every sampled prompt still gets all 5 dimensions x 3 judges x 3 arms). - rich_harness_lift.py: new --shuffle-seed (>0 => deterministic seed-shuffle of the processing order, applied after prompt load). Deterministic + resumable: same seed => same order, already-graded cells still skipped, so no rework. - autonomous_engine.py: PERDIM_SHUFFLE_SEED (env-overridable, default 20260716), passed only for exhaustive n=0 perdim-full closure jobs; bounded/batched board jobs keep their existing deterministic prefix semantics. - tests: perdim-full job passes --shuffle-seed; bounded/batched jobs do not. Verified on the real 78,719-prompt registry: shuffle is deterministic, and a 2,000-prompt shuffled prefix matches the full category distribution to within 0.4pp (vs the lumpier file-order prefix). 102 engine+grader tests pass.
…ign test) New self-contained grounded notebook on the real 85,417-row panel: per model (>=5 paired), the peer-review statistical battery recomputed from panel_grades.csv -- paired lift + seeded bootstrap 95% CI, leave-one-judge-out envelope (min/max lift dropping each judge), Cohen's d_z, exact two-sided sign test (renders <1e-300 on underflow, n/a when all tied), win rate + Wilson interval, and a cross-model summary table + forest plot. Honest "two limits" note (LLM judges, placebo confound). Built by a subagent, independently re-verified: 0 cell errors, and the headline Cohen's d = 1.73 (= 40.71/23.49) confirmed by hand from the raw panel.
The benchmark abstract claimed "headline models exceed Cohen's d = 3", but the board table directly below it shows the headline models at d ~= 1.3-1.9 (gemma4:31b 1.69, gpt-oss:120b 1.3); only qwen3.5:397b reaches 3.04 and it is a small-sample (n=40), low-lift breadth model, not a headline. Reworded to the honest, table-consistent statement: headline d ~= 1.3-1.9 (well beyond the 0.8 "large" threshold; d passes 3 only on the smaller-sample models), win rates 91-100%, p<0.001. Surfaced while verifying the new statistical-robustness notebook (Cohen's d_z = 40.71/23.49 = 1.73, confirmed by hand).
… values (drop stale d=3.47)
…t door) A judge-facing entry point that ties the Kaggle collection together: a rich banner + TOC, the real headline (recomputed live: gemma4:31b 48.4->89.1 +40.7), the cross-model board with normalized gain, a GUIDED TOUR table linking the grades dataset + reproduce + breakdowns + statistical-robustness + judge-agreement notebooks (in reading order) + the leaderboard dataset + repo + site, a reproduce section, and a prominent "what this does and does NOT prove" boundary. Everything recomputed from panel_grades.csv, nothing hard-coded. Verified via --execute-local: 0 cell errors, headline reproduces at +40.7 over 7,953 paired prompts.
…ks, boundary, slug)
New judge-facing notebook: an honest evidence ladder, each rung a claim + its support + its limit, grounded in the real 85,417-row panel. Rung 1 (STRONG) live paired lift +40.7; Rung 2 (STRONG) holds across judges; Rung 3 (MEDIUM) knowledge not preamble (placebo +3.34, notes the small study + full-registry placebo as the honest next control); Rung 4 (MEDIUM) citations real (~0% hallucinated); Rung 5 (honesty) sample sizes vary; Rung 6 (boundary) what it does NOT prove -- LLM judges not professionals, no victim-ID/field-detection, human-expert validation as precondition. Rich banner + TOC + collection cross-links (quality bar). Built by a subagent, independently re-verified: 0 cell errors, 0 banned patterns, 6 rung anchors, headline +40.7 recomputed.
…(claim-ladder + calibration)
…clusion) New grounded notebook (distinct from judge-agreement's ICC focus): judge leniency (glm-5.2 runs ~2.5pts hotter than deepseek), the paired lift recomputed within each judge (agree within 0.8pts, mean +40.7), averaging-reduces-noise (SD 25.2->23.5, honest caveat that the single lowest-variance judge beats the panel), and a live judge-family x model-family cross-tab proving self-exclusion holds (every same-family cell is 0). Rich banner + TOC + collection cross-links. Built by a subagent, independently re-verified: 0 cell errors, 0 banned patterns.
…-control / applicability) Packages the controls behind the harness-lift headline into a citable, scores-only dataset: placebo_panel (3-arm control scored by 5 self-family-excluded judges), negative_control_deterministic, applicability_audit, placebo_single_judge. A hard guard refuses any input with a >200-char field so response text can never leak. README states the honest findings including the inconclusive parts: the placebo PANEL closes the "any preamble helps" confound (+3.78..+5.52 beyond placebo, every judge), the deterministic negative control is honestly inconclusive (+0.08, p>0.05), applicability kappa=0.36. Propose-only; stages under reports/. Verified: 4 CSVs, scores-only guard passes, no PII.
Verifying the controls notebook surfaced two honest issues with the just-published dataset: (1) the placebo-panel harnessed-minus-placebo range recomputes from placebo_panel.csv to +3.34..+5.52 (panel mean ~+4.1), not the +3.78..+5.52/+4.55 copied from the richer jsonl analysis; corrected the README to the CSV-true values. (2) negative_control.jsonl is 100% the placebo arm (12,578/12,578), so the CSV cannot show the baseline->placebo->harnessed contrast; relabeled it honestly as the placebo arm's per-dimension deterministic scores (a ceiling diagnostic ~5.7/10), with the full contrast + the +0.08 result pointed to docs/research/negative_control.md. Re-versioned the public dataset. Core claim (positive for every judge, knowledge dwarfs preamble) unchanged.
The credibility notebook: recomputes the controls from the controls dataset, with honest framing throughout. Placebo panel (decisive) -- harnessed beyond a length-matched placebo +3.34..+5.52 for every one of 5 judges, panel mean ~+4.3, vs a generic preamble adding only ~+0.8; the confound is closed robustly to judge choice. Deterministic negative control -- framed explicitly INCONCLUSIVE (placebo arm parks near 5.66/10, harnessed-placebo +0.08 p>0.05; not evidence FOR the harness). Applicability audit -- grader/judge agree 68.2%, Cohen's kappa 0.36, 85.9% unanimous. Convergent validity noted as directional-only. Rich banner + TOC + collection cross-links. Built by a subagent, independently re-verified (0 cell errors); the agent also caught the 2 dataset-README errors fixed in the prior commit.
…set (8 notebooks, 4 datasets)
…u need?) A grounded methodology contribution on the published grades dataset: shuffle the headline model's paired prompts (fixed seed), then watch the mean-lift estimate + bootstrap CI and the category representativeness converge as the random subsample grows. Real result (verified via --execute-local, 0 cell errors): a random 100-prompt subsample already lands within 1.0 point of the full-panel +40.7, and the category-distribution gap shrinks with N. This is the methodological justification for reading the exhaustive per-dimension sweep at any interim point (a shuffled prefix is an unbiased random sample) -- interim goals reduce the prompt COUNT, not the grading resolution. Honest boundary: convergence of one estimate on one panel, not a licence to shrink the benchmark (tail categories/per-dimension detail still need the full sweep). Banner + TOC + collection cross-links.
…ll grader=perdim) [autonomous_engine] board data only; generated by scripts/autonomous_engine.py.
Fills the collection's biggest gap -- every other notebook is technical rigor; this grounds the WHY, from the real prompt metadata. Shows the exploitation typologies and named recruitment corridors the benchmark spans (128 typologies, 24 corridors, all difficulties in the grades dataset), then maps the five rubric dimensions to concrete worker protection (A name the indicator, B cite the law, C refuse to help the trafficker, D route to help, E protect privacy) with the mean per-dimension gain. Honest boundary: benchmark of response quality judged by LLMs, not victim-ID/field detection; composite scenarios, no PII; human-expert validation is the named next step. Verified via --execute-local, 0 cell errors.
…-tuning) Closes the loop from the benchmark to fine-tuning, scores-only (no response text): applies the real distiller gate (harness_core >= 70 AND lift >= 20, teacher arm = harness_core) to the published grades and shows how many benchmark prompts become training pairs. Real result (verified --execute-local, 0 cell errors): 5,980 of 7,953 paired prompts (75%) clear the gate for gemma4:31b -- each a ready SFT target (harnessed reply as gold) and DPO pair (chosen=harnessed, rejected=baseline), mean lift +49. Honest: judge-scored SILVER not gold, fail-closed on eval overlap, teaches structure not volatile facts, response rows live PII-scrubbed in the published training corpus, and NO trained model is claimed here. Banner + TOC + cross-links.
…otes the full sweep still completes
…nish) analyze_dips.py surfaces the tail the converged headline hides and turns it into an actionable training-data worklist, grounded in the real panel: - REGRESSIONS (harness < baseline): 15 prompts (0.2%) for gemma4:31b, worst on labor_trafficking (-11), prompt_injection_amplification (-7.3), and -- tellingly -- legal_accuracy (-5): the harness's own citations sometimes hurt legal accuracy. Action: exclude from SFT/DPO (anti-signal) + fix the harness for those categories. - VALLEYS (lowest-lift typologies): victim_revictimization (+11.6), bystander_dilemma (+12.0) vs +41 on labor_trafficking. Action: target with more/ better training data. - WEAKEST DIMENSION: C (refuse) at +6.7. Action: reinforce in the SFT/DPO targets. This is the concrete reason the exhaustive sweep must run to 100%: the dips live in the tail; a subsample misses them, and the complete dip set makes the next round of training data better. 3 tests; worklist -> docs/research/harness_dips.md.
… the live perdim panel each pass
The per-dimension (all-dimensions) sweep output (panel_perdim.jsonl) was local-only/gitignored. build_perdim_grades_dataset.py packages it as a scores-only (no text/PII, hard guard) Kaggle dataset: 0-100 + A-E component scores per (model, prompt, arm, judge), honestly labeled as a re-versioned growing snapshot. Closes the 'available/versioned' gap for the highest- resolution output. 2 tests.
Wire the newly-published duecare-harness-perdim-grades into the collection index notebook's guided tour (DS_PERDIM) and add a 'Published benchmark grades' catalog to training_and_finetuning.md so the exhaustive per-dimension output is documented as a model-agnostic train/test/benchmark label source. Public-surface audit clean; index + perdim tests green.
… materials New deep-reasoning tier above the 8-persona bundle: 101 real domain perspectives across 9 role categories (workers by sector/status, family, frontline support, origin/destination state, justice, supply chain, recruitment chain, observers), each producing a 102-step chain-of-thought that walks the full analysis grounded in real domain vocab (all 11 ILO forced-labour indicators, actor map, 7 journey stages, jurisdictions, evidence states, counterfactuals, role-bounded options, verification). Adds the reach (small/large jump) and direction (inward/outward) axes. Honest + safe: illustrative ILO-indicator patterns, no real individual/PII, propose-only, never framed as 'synthetic composite case'. 5 tests.
Make the 100+ perspective / 102-step CoT generator produce a real training dataset, separate from answer SFT/DPO: each row carries the 11 fields the executable training contract requires (id, lineage_id, split, source_profile, rubric_targets, source_refs citing ILO, quality_gate, contract-canonical sha256, license, pii_checked). Deterministic (situation x perspective) family holdout -> cot.jsonl (train) + cot_holdout.jsonl + cot_manifest.json, sharing one lineage space so nothing leaks across the split. build() validates the train stream against training_contract.validate_training_rows and fails closed; all 8 blocking gates pass (1740 train / 280 holdout). 6 tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
analyze_full_results.pynow joins the accumulated judged panel to the real 78,719-prompt registry — current readgemma4:31bbaseline 48.4 → harness_core 89.1 (+40.7, bootstrap 95% [+40.2, +41.2], win rate 99.8%) over 7,953 paired real registry prompts, consistent within each of three judges, largest lift on the hardest prompts, full − core = −1.04 at n=7,958 (serve core, not full). Full aggregate published as a JSON sibling; nothing truncated.prepared_not_executed, local evidence only, training-ineligible.Test plan
scripts/validate_public_surface.py— 8 checks, 0 findings (includes new fallback-registry check)python -m pytest packages --collect-only -q— 1,303 tests collectedscripts/validate_project_bible_pickup.py— 65 checks, 0 findings