Ground the flywheel studies in real recorded results#2
Open
TaylorAmarelTech wants to merge 97 commits into
Open
Ground the flywheel studies in real recorded results#2TaylorAmarelTech wants to merge 97 commits into
TaylorAmarelTech wants to merge 97 commits into
Conversation
…en judge audits Adds the post-merge Codex flywheel surface: Gemma 4 adapter study collection and notebook builders (six locally executed CPU notebooks), TPU LoRA training lab and grounded-scratch TPU notebook builders, measured-response curriculum builders, four-arm evaluation with recorded egregious-example join, frozen frontier-judge and recorded harmful-request judge studies, the DueCare system evidence receipt, and a governed model-fallback registry with validator and public-surface check. Study summaries now report exact two-sided sign tests and evidence-scale labels beside bootstrap intervals so anecdote-scale pair counts cannot be over-read; notebooks compute the same statistics from recorded verdict counts, add pair-level criterion consistency (adapter vs harness kept separate), and keep JSON artifacts strict (no NaN). The multiperspective bundle builder gains the four-way curriculum-focus expansion, lineage fields, minimal near-miss preference negatives, and a content-similarity gate. llm_generate omits non-default sampling parameters for Anthropic routes. The Kaggle publish and training-run payload directories are now gitignored.
…xonomy analyze_full_results now joins the accumulated judged panel to the real 78,719-prompt registry: per-model statistical strength (seeded bootstrap 95% interval, exact/normal-approximation two-sided sign test, Wilson win-rate interval), per-judge robustness (the paired lift recomputed independently inside each judge's verdicts), and lift breakdowns by real prompt category, corridor, and difficulty, with the complete aggregate published as a JSON sibling so capped tables drop nothing. Current read: gemma4:31b baseline 48.4 -> harness_core 89.1 (+40.7 [+40.2, +41.2]) over 7,953 paired real registry prompts, consistent within all three judges, largest lift on the hardest prompts, and full - core = -1.04 at n=7,958. build_recorded_harmful_request_expansion freezes every qualifying recorded Gemma failure (46 of 52 ranked prompts have complete recorded pairs) from the actual egregiousness ranker and rich-lift ledger into a judge-ready pack with 92 blinded two-order requests pinned by canonical hash. The pack is prepared_not_executed, local evidence only, and training-ineligible; the recorded-study verifier now accepts expansion manifests, and the four-arm join gains an allow_partial mode that still refuses empty packs.
Fixes the four-vs-five public dataset count and the split showcase table in the Kaggle index; states in the README that the frozen 32-verdict judge audit did not support a positive training-lift claim so the +9.67 harness result cannot be misread as a fine-tune win; adds the live-panel read (7,953 paired real registry prompts, +40.7 with interval and per-judge robustness) beside the checksummed receipt numbers; documents the exact-sign-test upgrade and the prepared 46-pair recorded harmful-request expansion pack, including its prepared_not_executed and local-only boundaries.
…re signaling Addresses defects from a multi-agent survey of the committed flywheel: - training_contract: the unsafe-advice gate cleared an operational-uplift phrase if any negation appeared anywhere in the answer, so a far-away refusal could launder an operational instruction. Negation must now sit within a bounded window of the operational match (_operational_advice_is_unmitigated). Scans the chosen/assistant answer only, never the preference rejected side. Strictly fail-closed; all release/bundle suites still pass. - A-00 kernel: the canonical release verifier is exec'd from a repointable source root with no hash pin. It now records the verifier module SHA-256 in the verification receipt (canonical_verifier_sha256) and fails closed when DUECARE_A00_EXPECTED_VERIFIER_SHA256 is pinned and does not match. - ollama_adversarial_flywheel: a whole-run failure (e.g. Ollama down) previously wrote a 'successful' manifest with zero candidates. The manifest now carries run_status + exception_failures, and the CLI exits non-zero on failed_no_candidates. - build_gemma4_adapter_kaggle_collection: the default run list named future regime runs that resolve(strict=True) turned into a hard crash. resolve_run_ directories now skips missing runs (recording their names) while still enforcing the two-completed-run minimum; the regime matrix is optional. First test coverage for this packager. - Removed a rule-60 display truncation (bounded[:480]) in the showcase notebook.
The docs/README/Kaggle-index assert a release-manifest SHA-256 and exact row counts per published dataset that were otherwise unverifiable prose. Adds: - configs/duecare/training/published_dataset_claims.json: committed single source of truth (5 datasets, each SHA + row counts + staged-artifact globs). - scripts/verify_training_dataset_claims.py: re-derives each SHA-256 from the gitignored staged/re-downloaded release manifests and re-checks row counts, failing closed on mismatch; a clean checkout reports each as published-only. All 5 claims verify against local artifacts. - validate_public_surface.py: a published_dataset_claims check that fails if a registry SHA drifts out of the training doc (clean-checkout safe; now 9 checks, 0 findings). - docs/training_and_finetuning.md: points at the registry + verifier so the (git_sha, dataset_version) reproducibility invariant is executable.
Owner
Author
Second pass — hardening from a 4-agent survey of the committed flywheelPushed two more commits ( Correctness / safety gates
Reproducibility / credibility
Verified, not changed: the safety/harness headlines (+9.67 recorded, +1.73/+0.18/+4.39 receipt) are over the real registry/ledger/gallery; only the adapter +0.15 is synthetic mechanism-scale (honestly labeled). No script pushes to Kaggle. The Gates: 92 changed-area tests pass · packages collect 1,306 · public surface 9 checks/0 findings · ruff clean on all new/edited files. |
Extends the training-dataset claims verifier per two review forward-pointers: - The adapter-study registry entry now pins the seven headline numbers that live as hardcoded prose in the study notebooks (+1.73 model judge, +0.18 deterministic, +4.39 adversarial, +9.67/6-of-6 recorded, 60 steps, 817,152 trainable params, +0.15 objective). verify_training_dataset_claims.py re-derives each from the staged system-evidence receipt / run metrics and fails closed on drift, so a re-run that changes a number can no longer leave the notebook prose silently stale. All 7 re-derive locally. - Adds an opt-in, credentials-gated --check-live flag that does a read-only 'kaggle datasets metadata' lookup to confirm each id resolves to a live public dataset. Off by default and never run unprompted; it makes no publish call. This closes the residual 'offline can only prove self-consistency, not live match' gap without any automatic Kaggle traffic.
Implements the central recommendation of the 2026-07 harness/finetuning blueprint: turn DueCare's hard-coded intelligence into a governed, auditable supervision fabric that is never mistaken for ground truth. - duecare.chat.rulecards: compiles each raw GREP indicator rule into a typed RuleCard (authoritative sources, antecedent, consequence, inferred role, jurisdiction, witness family, calibration gaps). Every rule classifies as a fallible labeling_function + feature_extractor; none are auto-promoted to a hard invariant (a match is grounds for inquiry, not proof). Critical rules are flagged for human invariant review, never silently. - independence_report measures the blueprint's load-bearing insight from the REAL deck: 451 rules resolve to 80 correlated-witness families; the largest holds 86 rules and the top five hold 63% of the deck (Palermo 153, ILO C181 113, C029 89). Same-family rules are correlated votes, not independent confirmations — the input a correlation-aware weak-supervision label model needs so one legal principle expressed as many patterns cannot inflate confidence. - scripts/build_rulecards.py compiles the live rules + their CATEGORY headers into a deck, summary, independence JSON, and a markdown report (deterministic, no model call; output gitignored). - docs/research/rulecard_supervision_fabric.md reconciles the full blueprint against DueCare's measured state (451 rules, +40.7 real-panel harness lift, the honest fine-tune result) with a build order grounded in what exists. 9 module tests; packages collect 1,315; public surface 9 checks/0 findings.
The showcase notebook quotes +1.73/+0.18/+4.39/+9.67 as narrative prose while its charts read the same numbers from JSON. The load cell now re-derives all four from the packaged system-evidence and recorded-safety receipts and raises if any drifts, so a re-run that changes an artifact can never leave a stale hardcoded figure in the narrative. Executes clean (0 errors) against the real receipts.
Owner
Author
Third pass — RuleCard supervision fabric + blueprint reconciliationBuilding on the harness/finetuning deep-research blueprint, with everything grounded in the real repository (per the standing "avoid too-synthetic" directive). RuleCard supervision fabric (the blueprint's central recommendation)
Claim provenance (both reviewers' forward-pointers)
Gates: 78 changed-area tests pass · packages collect 1,315 · public surface 9 checks/0 findings · ruff clean · study notebooks re-execute with 0 errors. Note: |
Extends the RuleCard fabric with the blueprint's correlation-aware effective witness count and packages it for publication: - rulecards.effective_witness_count applies the design-effect formula m/(1+(m-1)*rho) per witness family and sums across families, so the structural 80-family count (rho=1 bound) is shown alongside softer correlation assumptions. On the real deck: ~83 effective witnesses at rho=0.9, ~90 at 0.7, ~100 at 0.5 -- far below 451 raw rules. independence_report now carries effective_witnesses_by_rho with an honest bounds note. - scripts/build_rulecards_kaggle.py packages the deck as a public CC-BY-4.0 Kaggle dataset (typed deck, independence report, summary, flat CSV, data card, license, SHA-256 release manifest) plus a CPU visual notebook that renders the reduction result, effective-witness curve, per-instrument concentration, severity/review boundary, and category coverage. Rule metadata only -- no worker data or PII. Locally executed clean (0 errors, 4 charts).
The RuleCard supervision-fabric dataset is now live and public at taylorsamarel/duecare-rulecard-supervision-fabric (CC-BY-4.0, 451 RuleCards, 80 witness families, effective-witness range). Sets isPrivate:false on the dataset metadata and gives the companion visual notebook a distinct id/title (duecare-rulecard-witness-atlas) so it no longer collides with the dataset slug.
Owner
Author
Fourth pass — effective-witness math + first Kaggle publishBuilding on the full deep-research blueprint, with Taylor's explicit authorization to publish to Kaggle this session. Effective independent-witness count (blueprint §2.2) Published — live and public on Kaggle
Gates: 11 RuleCard tests pass · ruff clean · public surface 9 checks/0 findings · dataset PII scan clean (rule-name matches like Six commits this session's later waves; branch pushed. The RuleCard dataset is the first genuinely novel, honest, public artifact from the supervision-fabric direction. |
…e notebooks Turns the thousands of real Ollama/multi-source gradings into a public research asset and reproducible example notebooks -- maximally using the graded panel. - build_benchmark_results_kaggle.py packages the real 85,417-row graded panel (0-100 rubric score + A-E component scores per model/arm/prompt/judge) as a CC-BY-4.0 Kaggle dataset with SCORES ONLY -- no response text, no prompt text, no PII -- plus joinable prompt metadata (category/corridor/difficulty labels) and the precomputed harness-lift analysis. Rich metadata for usability: subtitle, 200+ char description, keywords, and per-column schema descriptions. - Two runnable example notebooks (attach the dataset, no internet): 'Reproduce the Harness-Lift Result' recomputes the +40.7 lift from scratch with bootstrap CI, exact sign test, Wilson win-rate interval, per-judge robustness, and category breakdowns; 'LLM Judge Agreement' measures pairwise judge correlation (0.905-0.925 on the real grades) and per-component score patterns. Both locally executed clean (0 errors, 5 charts). - 3 tests: prompt-text-never-leaks safety, usability metadata + column schema, and runnable notebook contract.
…on, column schema) Both DueCare datasets now carry full usability metadata: a <=80-char subtitle, a 1000+ char description, four valid-vocabulary keywords, per-file descriptions, and per-column schema descriptions for every CSV. Kaggle scores these directly. Constraints learned and encoded: keywords <=6 (used 4 valid slugs), subtitle 20-80 chars. Both datasets republished public.
Kaggle derives a kernel slug from its title; a title that does not slugify to
the metadata id fails the push ('title does not resolve to the specified id').
The example-notebook titles are now bare 'DueCare Reproduce Harness Lift' /
'DueCare Judge Agreement' so they slugify to exactly their ids.
…s read analyze_full_results now cites the public grades dataset + reproduction notebooks in the generated report header (the report is regenerated, so the link belongs in the generator, not the .md). Also adopts the datetime.UTC alias via linter autofix.
…d deck Dual-use safeguard. The public rulecard-deck.json exposed the exact regex patterns for all 451 rules -- an operational detection map that could help offenders rephrase to evade it. The project's safety rules and the finetuning blueprint both forbid publishing evasion-enabling operational detail. The public deck now carries pattern_count (not the raw regex) per card while keeping all research metadata (legal sources, severity, roles, jurisdictions, witness families, indicator reasoning); the redaction is disclosed in the README. The full-pattern deck remains available internally for DueCare's own use. 3 tests guard against regex leakage.
Makes DueCare launchable in one step for NGOs, developers, researchers, and notebook users, and fixes a real containerized-launch bug found while reviewing the deployment surfaces: - FIX: the Docker ENTRYPOINT was 'python -m duecare.chat.run_server', but run_server was only a FUNCTION -- no such module existed, so the container crashed on start. Adds the missing runnable module packages/duecare-llm-chat/src/duecare/chat/run_server.py (a thin argparse CLI around the existing run_server()). 'python -m duecare.chat.run_server' now works; the image starts. - launch.py: one friendly launcher mapping audience -> command (workbench / ngo / demo / benchmark / notebook), with prereq detection and --dry-run. 6 tests. - docker-compose.build.yml: a build-from-source override so a fresh clone can 'docker compose up --build' without needing the published GHCR image (verified valid with docker compose config). - docs/QUICK_LAUNCH.md + a notebook launcher (examples/deployment/notebook/) that runs the server in Colab/Kaggle/Jupyter and exposes a public tunnel URL. - README quick-start now points at the one-command launcher.
Implements the confirmed methodology findings from a PhD-stats review: - Leave-one-judge-out lift envelope (the judge-averaged bootstrap marginalizes out judge variance, so [+40.2,+41.2] was false precision). The envelope [+40.4,+40.9] is now the primary uncertainty statement; the prompt-only bootstrap is explicitly flagged as judge-blind. - Normalized gain (harness-baseline)/(100-baseline) controls for the rubric ceiling that mechanically compresses lift at high baselines (0.74 of headroom). - The sign test no longer renders 'p = 0.0' -- it shows '<1e-300' and is labeled direction-only; the win-rate Wilson interval carries a design-effect caveat (the same 3 judges recur across prompts). - 'serve core, not full' is now gated on a paired bootstrap CI that excludes zero; a ~1-point mean within noise is called statistically indistinguishable, never an operating rule. - The rubric-preamble/judge-rubric congruence confound (part of the raw +40.7 is instruction-following, not domain value) is surfaced in the report body with the length-matched placebo named as the next full-scale measurement. 5 new/updated tests. Direction remains well-supported and judge-robust; the magnitude and its uncertainty are now defensible as stated.
…reement Implements the confirmed prof-stats / eng-industry persona findings against the real 85,417-row grades panel (verified by --execute-local): - Judge-agreement notebook now reports WITHIN-ARM agreement (ICC(2,1) per arm + Fisher-averaged within-arm Pearson r) instead of pooled Pearson. Pooling across arms inflated r to 0.918; the honest within-arm r is 0.723, and ICC reveals judges agree strongly on baseline (0.80) but only moderately on the richer harnessed responses (0.42-0.45). The closing note no longer overclaims "high agreement" and points to the near-unanimous lift direction instead. - Both notebooks use a single canonical headline_model() (prefer gemma4:31b, else most-paired) so the lift and agreement cells never disagree on the head. - find_dataset() verifies the release-manifest dataset_id (binds to the right dataset even if another DueCare dataset is attached). - Sign test / Wilson interval guard the all-tied (info==0) and empty (n==0) cases; sign p renders as <1e-300 rather than a literal 0.0. - Lift cell raises a clear error instead of an IndexError when no model yet has >=5 paired prompts. Both notebooks execute end-to-end against the real panel: headline reproduces at gemma4:31b 48.4->89.1 (+40.7), sign test <1e-300 over 7,938 wins / 15 losses.
Applies the confirmed eng-industry findings to the public RuleCards supervision-fabric notebook, verified by --execute-local (0 cell errors): - find_dataset() now verifies release-manifest.json dataset_id against the expected slug, so the notebook binds to the right dataset even when another DueCare dataset is also attached (was: first subtree with the sentinel file). - Category chart filters 'uncategorized' BEFORE taking the top 16, instead of slicing first and then dropping it -- the old order silently lost a real category whenever 'uncategorized' fell inside the top 16. Title now reflects the actual named-category count. Rebuilt + executed against the real 451-rule / 80-family deck; effective witnesses 82.7-100.1 across rho, no cell errors.
… claims Domain/legal-expert review of the vetted claim library (85 claims) surfaced two overstatements that the freshness validator flagged: - sa_kafala_reform_2025: an advocacy-sourced (Walk Free), pending-human- verification announcement was tagged binding_domestic. Reframed the text to attribute the "end of kafala" to reporting and foreground that it is a REPORTED, partial reform (sponsorship persists in modified form, domestic workers largely excluded, prior Gulf promises outran implementation), and downgraded binding_status -> reported_reform_unverified. - eu_forced_labour_regulation: binding_status binding_domestic could let a consumer that reads only the status (not the caveats) present a rule whose market-ban obligations do not apply until 14 Dec 2027 as currently enforceable. Encoded it as enacted_not_yet_applicable. Both claims remain flagged for recheck (they still need primary-source human verification); this only makes any grounded use before then honest. Library still validates (85 claims, 13 tests pass); register regenerated.
…notebooks Kaggle's SaveKernel has a per-account DAILY push cap (resets 00:00 UTC); once hit, every `kernels push` returns 429 until the reset. This script makes publishing the corrected notebooks (judge-agreement, reproduce-harness-lift, rulecard-witness-atlas) re-runnable and scheduler-friendly: - stages each at a SHORT path + pre-creates the %TEMP% upload mirror (Kaggle CLI 2.2.x temp-path bug workaround), - distinguishes 429 (exit 3, "try after reset") from a genuine error (exit 2), - --wait-for-reset sleeps until just past 00:00 UTC then pushes, - --rebuild regenerates from the live panel/deck; --dry-run stages only, - verifies each slug against the account's live kernel list. Dry-run confirmed it finds + stages all three built notebooks.
The full-registry per-dimension sweep is a ~24-day-per-model job, so we want statistically valid INTERIM readouts long before it finishes. The promptset is built in a structured order (curated/stable head, then a stratified expansion), so a file-order prefix is a biased slice. This adds an opt-in seeded shuffle so any interim prefix of graded prompts is an unbiased random sample of the full registry -- interim goals reduce the prompt COUNT, never the grading resolution (every sampled prompt still gets all 5 dimensions x 3 judges x 3 arms). - rich_harness_lift.py: new --shuffle-seed (>0 => deterministic seed-shuffle of the processing order, applied after prompt load). Deterministic + resumable: same seed => same order, already-graded cells still skipped, so no rework. - autonomous_engine.py: PERDIM_SHUFFLE_SEED (env-overridable, default 20260716), passed only for exhaustive n=0 perdim-full closure jobs; bounded/batched board jobs keep their existing deterministic prefix semantics. - tests: perdim-full job passes --shuffle-seed; bounded/batched jobs do not. Verified on the real 78,719-prompt registry: shuffle is deterministic, and a 2,000-prompt shuffled prefix matches the full category distribution to within 0.4pp (vs the lumpier file-order prefix). 102 engine+grader tests pass.
New batch (non-server) kernel A-30: installs A-00's proven Unsloth stack, LoRA-fine-tunes gemma-4-E2B (4-bit) on the published duecare-cot-reasoning rows on Kaggle GPU, does before/after generation on a held-out prompt, and saves the adapter + run evidence. enable_gpu + internet, attaches the CoT dataset, no HF token needed. Pushed + running as taylorsamarel/duecare-cot-gpu-lora-training.
Per Taylor 2026-07-20: agent may actively publish and run Kaggle kernels (incl. GPU/TPU training) without asking; removes the manual-by-default / user-driven / no-REST-push constraints. Safety + PII gates still apply.
…rch mismatch on T4
…agnostics Root cause of cudaErrorNoKernelImageForDevice was force-installing torch>=2.8 from PyPI, replacing Kaggle's driver-matched torch with a generic wheel. Now pin torch to the already-installed version so pip keeps it while resolving unsloth deps; print sm/arch_list to confirm.
…r T4 Root cause: Kaggle assigned a Tesla P100 (sm_60), but torch 2.10's arch_list is sm_70+, so no kernel image. CLI metadata cannot pick the GPU, so exit cheaply on an unsupported card and re-run to draw a T4 (sm_75).
….8% win rate) Visual single-story showcase rendering matplotlib charts from the published grades dataset: lift distribution, cross-model leaderboard, per-dimension A-E, convergence, lift-by-difficulty. DueCare palette, honest boundary. CPU-only, runs to completion on Kaggle. Local validation caught + fixed a conflated win rate (73% was a different benchmark; the +40.7 board is 99.8%, 15 hurt).
…ransformers/peft, no Unsloth) Kaggle assigns P100 (sm_60); modern torch/Unsloth dropped it. Install torch 2.5.1+cu121 (includes sm_60) + plain transformers/peft/trl, LoRA-train gemma-4-e2b (fallback gemma-3-1b) on the CoT dataset. Verifies sm in arch_list, model fallback so the run always completes.
…ch fix confirmed working
…odel, no image processor)
…mma3n on P100, loud fail, no Gemma-3)
… distributions, dimension heatmap, judge agreement, category breakdown, TOC, rich markdown
… like A-dot / a-em-dash)
…dditive-dimension finding, judge comparison)
…ks, load snippet) + nav link
…n /kernels + link /data
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
analyze_full_results.pynow joins the accumulated judged panel to the real 78,719-prompt registry — current readgemma4:31bbaseline 48.4 → harness_core 89.1 (+40.7, bootstrap 95% [+40.2, +41.2], win rate 99.8%) over 7,953 paired real registry prompts, consistent within each of three judges, largest lift on the hardest prompts, full − core = −1.04 at n=7,958 (serve core, not full). Full aggregate published as a JSON sibling; nothing truncated.prepared_not_executed, local evidence only, training-ineligible.Test plan
scripts/validate_public_surface.py— 8 checks, 0 findings (includes new fallback-registry check)python -m pytest packages --collect-only -q— 1,303 tests collectedscripts/validate_project_bible_pickup.py— 65 checks, 0 findings