release: v0.0.33 - #139
Merged
Merged
Conversation
chore: back-merge main into develop after v0.0.32
Confirmation criterion met on the v0.0.32 iris run: stacking AUC within +/-0.002 of the boruta baseline across all six models, raw-feature models better for the in-context learners, grootcv + name sanitization + GPU ablation routing all verified in production. Default flipped in the NF param, both CLI commands, and multimodal_prep/multimodal_eval; the low-level _select_multimodal_features primitive keeps mi. Outside the shipped containers the default path now needs kreview[arfs]. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…blations, no false leakage banner Four defects found reviewing the v0.0.32 iris report: 1. report.nf staged every model-results JSON into models/cpu/, but build_report_data reads models/gpu/<eval>_gpu_model_results.json — the modal silently showed only lr/rf/xgb. Staging routes by filename now; structural guard in the stub test. 2. The multimodal tab embedded LOO ablation data but only rendered the absent-note — no success-case code existed. Added the contribution bar chart with loud-fallback annotation. 3. multimodal_ablation minted phantom evaluators from two-part model suffixes (X_tabicl_ft -> phantom X_tabicl): 52 ablations instead of 26, half the LOO GPU compute wasted. Discovery strips known model suffixes; the report filters phantoms from pre-fix files. 4. The patient-leakage counter grouped over every split value, so (exclude, train) patients false-flagged the banner (14 on v0.0.32 with zero true leakage). Counts train/test rows only now. Verified by re-rendering the real v0.0.32 outdir: 7 models in the modal, 26 clean deltas, patients_in_both_splits=0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
feat(multimodal): #96 closeout — flip selection default mi → grootcv
R.meta.generated_note does not exist (meta = run/title/version), so the header ended in '· undefined'. Replaced with the always-true PHI note. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two additions requested from the v0.0.32 review: - Each tab opens with a collapsible 'What this tab computes' block stating exactly what the numbers are (5-tier labeling + grouped split, mRMR + OOF metrics + holdout semantics, stacking columns + LOO granularity, trace parsing). - _derive_findings: a deterministic rules engine in the DATA layer that turns the run's own numbers into tab-tagged findings (split integrity, evaluator health, overfitting check, weak evaluators, GPU dominance, healthy-anchor variance, stacking lift vs best-single CI, meta-learner spread, LOO concentration/redundancy, retry attribution) rendered as a Key-findings panel on the overview tab. Aggregates only; PHI-guarded with the rest of the payload; testable. On the real v0.0.32 run this derives 12 findings, e.g. stacking +0.048 beyond the best single's 95% CI, 90% of unique LOO contribution in FSCGenomewide/TfbsOnTarget/FSC_gene, 23/26 evaluators individually redundant. Verified live in a browser (all tabs, zero console errors). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The export-sync gate caught it — _derive_findings entered the module index but the file was not staged with the feature commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…msk-access/kreview into fix/report-gpu-models-loo-panel
fix(report): GPU models in deep-dive, LOO panel renders, no phantom ablations, no false leakage banner
…admap Six reproducible offline analyses (stacking granularity, detection-set complementarity, tumor-informed true-negative anchoring, lead-time, clean-label tier profiling, assay-effect battery). Aggregates only. Findings live on the research-roadmap issue. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the reproducible set (10 scripts). Their OUTPUT workbooks carry sample identifiers and stay local — warning added to the README. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- scripts/README.md: one-line index of every guard/ops/analysis script - CONTINUITY.md: rewritten from the stale July state to the current one (v0.0.30-32 shipped, research campaign findings, in-flight items) - memory: project-research-roadmap (settled conclusions + pointers); identifier-bearing workbook locations recorded in the PRIVATE store only Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chore(analysis): research-analysis scripts behind the post-v0.0.32 roadmap
…ta firewall) From the adversarial-review triage (#121): declares the confirmatory primary (sens@98spec-TN, tabicl_ft, a priori), secondary endpoints, the exploratory boundary, and standing rules — metadata is evaluation/ calibration-only and never a model feature; donor-anchored sens@100spec is a one-sided bound; pooled AUCs ship with the verified-TN number. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
docs: pre-registered analysis plan
…data and the label bug it exposed (#122) #123 — the report now leads with clinically meaningful operating points: - sens@98/99% specificity against tumor-informed true negatives (MRD question), computed from the PRE-REGISTERED score (multimodal stacking, PRIMARY_MODEL) when the stacking matrix is available; the score used is always named, with an explicit fallback chain, never silently mixed. - donor-anchored sens@100spec demoted to a ONE-SIDED lower bound: its threshold is a max-statistic over ~55 donors, for which a symmetric bootstrap CI is inconsistent by construction. - verification-bias triple (all negatives / verified TN / donors only), PPV at stated prevalences, and a winner's-curse annotation showing the a-priori primary evaluator beside the run's argmax. Verified on the real v0.0.32 outputs: 0.4834 @98% spec-TN, 0.343 @99%, donor bound 0.685, AUCs 0.8556/0.8452/0.9757 — matching the independent analysis exactly. #122 — implementing depth covariates exposed a latent production bug: the 'kreview label' command never accepted --krewlyzer-dir (it built Paths(..., [])), so metadata never loaded, total_fragments_pf was never computed, and the configured, printed, documented --min-fragments Insufficient-Data rule has never fired in any run. Fixed: the option exists; label.nf, both workflows and main.nf pass it (sentinel-tolerant on the label-only path); the column is ALWAYS emitted (NaN = explicit unknown) and registered in LABEL_META_COLS so the ANALYSIS_PLAN metadata firewall holds; a missing depth source logs depth_metadata_unavailable naming the impact; a structural stub-test guard prevents the wire from regressing. Behaviour change: runs supplying the krewlyzer dir will now correctly label low-depth, evidence-free samples Undetermined — counts shift slightly vs <= v0.0.32. Verified: 393 tests (3 failures are the known local arfs-2.4 artifact, green in the arfs-3.0 env and in CI's test-strategies job), export idempotent, black/ruff clean, CI-exact container mypy clean, stub test passes on the v25 floor with the new guard, v26 config parses, PHI and version gates pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The data tabs had accumulated four collapsible methodology blocks plus inline notes, and the prose was competing with the numbers. Static explanation now lives in one 'Methods & interpretation' tab (definitions — including the qualified-true-negative rule and why the donor number is a one-sided bound — plus per-tab methodology and the honest-reading notes); run-derived findings stay on the overview where the results are. An in-page link points there; showTab hardened to highlight the nav when called without a button. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI runs at 80 columns, where rich wraps the flag across lines; normalize ANSI + whitespace before asserting. The option itself was correct. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-off drawn Visual-design review slices 1-2. The report had no focal point (the headline number was a table cell with the same weight as 'Longest tasks') and omitted its own interpretive key: the detection-vs-tumor- burden curve existed only in offline analysis scripts. - Hero zone answers the three arrival questions: how good (primary endpoint, large, with anchor + 99% companion), at what burden (LOD curve, Wilson CIs per VAF bin, LOD50 interpolated in log-VAF and reported in tumor fraction too), does combining help (stacking lift with meta-learner spread). - Specificity/sensitivity trade-off is now a curve with the declared operating points marked and the donor value drawn as a one-sided arrow at 100% spec, so the max-statistic caveat is visible rather than a footnote. Data layer: LOD_BINS, _wilson (correct coverage at the tiny-p, small-n end where the normal approximation breaks), sens_spec_curve and lod blocks on the operating points; all PHI-guarded aggregates. Verified on the real v0.0.32 outputs in a browser: marks at 0.4834 / 0.343, donor arrow 0.6849, LOD bins matching the offline analysis, zero console errors. Tests assert monotonicity both ways (sensitivity falls as specificity tightens; detection rises with burden), Wilson bounds, and the TF = 2 x VAF relation. CI-exact container mypy clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n-width) Reported: the trade-off plot in Cohort & labels spilled outside the white panel. Cause: grid items default to min-width:auto, so the operating-points table (th/td are white-space:nowrap) forced its column to min-content width and pushed the chart column past the panel edge. - .grid2 > * and .hero > * get min-width:0 so columns may shrink; the table scrolls inside its own cell instead of widening the row. - the ops row uses explicit minmax(0,...) tracks and the plot div is width:100%. - plots are resized when a tab is revealed (a chart drawn in a hidden tab has no measurable container width and gets no resize event), and a ResizeObserver now tracks every plot container, so a chart can no longer keep a stale width in any container — grid cell, hidden tab or dialog. Verified by static assertion on the rendered page (all seven guards present); browser verification was unavailable this session (pane not compositing) and the sandbox lost read access to ~/Downloads, so please confirm visually. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…o-ops Follow-up to the grid fix; the real cause went deeper than CSS. - Plotly.Plots.resize() returns without error but leaves _fullLayout.width untouched for charts whose container was narrower than plotly's 700px default at draw time, so their SVG kept overflowing the panel. Replaced with fitPlot(): measure the container and Plotly.relayout explicitly. - Column sizing for the operating-points row moved from an inline style to a .grid2.ops class — an inline grid-template-columns outranks the stacking media query, so the row stayed side-by-side on narrow viewports and pushed its own panel past the screen edge. - nav wraps instead of forcing horizontal page scroll on small viewports. - fitPlot runs on registration, after paint (rAF x2), on load, on tab reveal, and from a ResizeObserver per plot. Verified in-browser on the real v0.0.32 outputs at 1265px: zero page horizontal overflow and zero oversized SVGs on all five tabs and in the deep-dive modal (4 charts). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cells inherit white-space:nowrap globally — right for the dense numeric tables, wrong for these prose rows, which overflowed into a scroll container so the anchor column sat out of view. The ops table now wraps its cells and needs no horizontal scroll (table 617px = container 617px, no clipped cells, verified in-browser). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…harts Slices 3-4 of the visualization review. Grammar (slice 3): - AUC is drawn identically everywhere: dot + 95% CI whisker on a shared axis. The evaluator ranking was a bar chart — bars imply a meaningful zero and hide uncertainty, both wrong for a discrimination measure — and now carries CV dots with CIs plus holdout diamonds. - Three disjoint colour scales, so a hue means exactly one thing: CATEGORICAL (series identity, excludes every status hue), STATUS (good/warn/bad, always paired with a glyph so it survives greyscale and colour-blind vision), BURDEN (sequential, reserved for tumour burden — now used for the LOD markers). The multimodal 'raw' series was green, colliding with status-good; it is categorical now. The donor caveat mark keeps the status hue deliberately (it annotates a caveat, not a series) and gains a text label so it is never colour-only. Assertion titles (slice 4): - Findings can name a chart they belong on; those sentences become the chart title (wrapped), and the findings list keeps only claims with no chart home — 16 items became 13, each remaining claim sitting on its own evidence rather than in a separate wall of text. Verified in-browser on the real v0.0.32 outputs: titles carry the derived claims, the evaluator chart is two marker traces, no status hue is used for a series, and every tab still has zero page overflow and zero oversized charts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… labelled
Three readability defects, all in the single-page template.
1. Titles overlapped the plots. PL_LAYOUT used a fixed top margin, so any
title that wrapped to two or three lines (the assertion-style titles from
claimTitle do, routinely) drew straight through the top of the trace area.
The margin is now derived from the title's line count (12 + 23 per line),
and a caller-supplied partial margin is merged rather than replacing it,
so passing {l:...} no longer silently drops the computed top.
2. Domain terms had no in-place explanation. A 16-entry GLOSSARY now backs an
ii(key) helper that emits a keyboard-reachable info icon; clicking one opens
a popover with the definition and, where there is one, the upstream
reference. 18 icons are placed across the overview, scoreboard and
multimodal tabs. The scoreboard's click-to-sort headers ignore clicks that
land on an icon. The glossary block sits immediately after the esc helper
because the hero and operating-point IIFEs call ii() as they build markup.
3. The stacking-vs-raw chart labelled only half its rows. Twelve categories
(six models x two scopes) in a 280px panel made plotly thin the tick labels
to every other one, which left six dots anonymous. Ticks are now forced to
one per row and the panel has the height to hold them.
Verified against the v0.0.32 iris outputs: all chart titles clear their traces,
all twelve rows are labelled, the popover opens and dismisses, and the console
is clean.
…evaluator
Two defects in the evaluator chart, one of which was hiding the other.
The chart draws 26 evaluators in a 340px panel, so plotly thinned the category
ticks to every other one — and the row it dropped was FSCGenomewide at the top,
the best evaluator and the whole point of the panel. Ticks are forced one per
row and the panel now has the height to hold 26 of them.
Fixing that exposed the second: PL_LAYOUT merged the caller's extra layout with
a shallow Object.assign, so passing {xaxis:{range:...}} replaced the entire axis
object and took its title with it. Three charts had silently lost an axis label
this way — the evaluator chart's "AUC" and both axes of the burden curve and the
sensitivity-specificity panel. Nested layout objects (title, axes, legend, font,
margin) now merge one level deep, so a partial override composes instead of
erasing. The evaluator legend moved down to clear its restored axis title.
Verified against the v0.0.32 iris outputs: 26 of 26 ticks drawn, every chart
reports its axis titles, and no title, legend or axis label overlaps or clips.
feat(report): hero zone with the burden-response curve, and the trade-off drawn
The new notebook was written as nbformat 4.4 with no cell ids, so nbdev derived a fresh cell hash on every export and `nbdev-export` never reached a fixed point -- the export-sync gate would have failed on the next unrelated run. Cell ids are now content-derived and the notebook declares 4.5, matching every other notebook here.
review_round3.py answers the four reporting fixes and two analysis requests in the reviewer's record, and three of them come back with different numbers than the reviewer assumed. Patient multiplicity: singletons are 73.9% of patients and 49.0% of samples, not the 49%/33% stated -- the conclusion (stop quoting 1.51 samples/patient, quote the exposure) is adopted with corrected figures. The leakage delta rescaled to the exposed subset SHRINKS rather than grows: -0.0007 [-0.0022, +0.0008] against -0.0010 pooled, an order of magnitude under the predicted 0.009 worst case. The exposed subset is intrinsically harder (AUC 0.8225 vs 0.8479) but grouping does not move it. Sensitivity resolution floor measured at 0.0216, not the assumed 0.043; the 'no shift detectable at this resolution' framing is adopted at the real number. Label-tier composition does not explain the histology gap: restricted to True+ only, NSCLC 36.1% vs pancreatic 20.2%, against 34.1% vs 21.5% unrestricted -- though pancreatic being 47.5% True+ against NSCLC's 75.7% is worth reporting. Histology enrichments now carry counts and Wilson intervals. Germ Cell Tumor survives (18/72, lower bound 16.4% against a 2.09% base rate) and Small Cell Lung Cancer joins it; the retinoblastoma OR 0.11 was a denominator artifact -- it has under 20 genotype-silent samples -- and is withdrawn.
feat(report): the pipeline DAG as a drawn object, in Methods and Run diagnostics
… root review_round3.py wrote its .npz cache to the repository working tree, so a run left an untracked 200KB artifact behind that anyone could commit by accident. It now caches inside the output directory it derives from.
fix(analysis): cache OOF predictions beside the data, not in the repo root
The file still described PR #120 as in flight and predates three review rounds, two releases' worth of merged work, and the finding that the headline arm cannot be tested on this hardware. Rewrites it as a register split the way the work actually divides: ship what is built, make the report admit what the review forced us to concede, research that runs on CPU here, research that needs the cluster, writing and decisions, and infrastructure. Each item carries why it exists and what gates it, because the gating is the useful part -- the F7 chain is four items deep behind one cheap re-label, and the headline's leakage answer is one GPU job short of complete. Also records the isolated-environment rule that a base-env install earned today: pip pulled setuptools past 81 and removed pkg_resources, which is exactly the breakage the setuptools<81 pin exists to prevent.
…pper Four small items from the post-review register, all of them things that bit us once already. F3 — notebook hygiene tests. A notebook written as nbformat 4.4 carries no cell ids, so nbdev derives a fresh export hash every run and `nbdev-export` never reaches a fixed point; the sync gate then fails on whichever unrelated PR next touches a notebook. That happened on the diagram module. The tests assert 4.5, unique ids on every cell, and that a notebook with export cells names its module. F4 — the two mypy errors that reproduce locally under the pinned toolchain but not in CI. Neither was type noise. `ndarray == str` narrows to `Any | bool` and bool has no `.sum()`, so count_nonzero states the intent instead. The cli one was pointing at a real fragility: pivot_table returns a flat Index when pandas collapses a single value column, and `for col, rt in pivoted.columns` would then unpack the characters of a column name — now guarded on MultiIndex, and the `type: ignore` is gone rather than widened. F1 — an sbatch wrapper for GPU research scripts that are not Nextflow stages (the tabicl_ft leakage arm is why it exists). Mirrors `withLabel: process_gpu`: same container, partition and time ladder, the same cache redirects the pipeline applies because /home is read-only on the compute nodes, and it fails loudly if no GPU is visible rather than silently finishing on CPU. F2 — an isolated-env recipe that refuses to install unless the target really is isolated (pyvenv.cfg present and sys.prefix != base_prefix). That check is what was missing when a scratch path whose python symlinked to the base interpreter took a torch install into the base env and removed pkg_resources with it. Also commits the tabicl_grouped_cv.py arm the wrapper is built to run.
docs: rewrite CONTINUITY as the post-review work register
chore: guards that keep the toolchain reproducible, and a GPU job wrapper
…width comes from The headline — sensitivity at 98% specificity against verified true negatives — shipped as a bare point estimate. `_wilson` existed but was used only for the LOD bins, so the number the whole report is built around carried no uncertainty at all. It now carries a patient-clustered bootstrap interval with the threshold re-estimated inside every resample. Both parts matter and a binomial interval on sample counts gets neither: repeated timepoints from one patient are not independent, and the threshold is itself an estimate from a finite negative set that moves when that set is resampled. Measuring it separated two things the review had conflated. On the v0.0.32 run, sens@98spec is 0.483 [0.424, 0.519], and decomposing the width gives a clustering design effect of 1.26 — squarely inside the reviewer's 1.14–1.23 estimate, now measured rather than assumed — against a threshold-estimation inflation of 11.3x. The term everyone has been arguing about is the smaller one by an order of magnitude. The report states both, next to the numbers they explain. Two things this exposed, both fixed here: The TN-anchored block only computes above 200 verified negatives, and the test fixture had 25 — so the report's primary endpoint had never been exercised in CI. The fixture builder is now parameterised and `anchor_outdir` clears the gate, with internally consistent label tiers (the small fixture's inconsistency is deliberate and several tests depend on it, so it is untouched). Dropping PATIENT_ID raised a bare pandas KeyError three frames down. It is a required column — the grouped split, the leakage check and now the interval all key on it — so the failure now names the column and says what breaks without it. The anchored block itself still degrades rather than crashing, recording `ci_method: unavailable` instead of substituting a narrower interval.
`_bootstrap_auc` resampled rows, so every evaluator and holdout interval in the scoreboard treated a patient's repeated timepoints as independent observations. That is the same defect just fixed on the primary endpoint, in the code path that produces every other interval the report shows. It now resamples whole patients when patient ids are available, and `patients` threads from the matrix through cpu_models/gpu_models into evaluate_model, _compute_oof_metrics and evaluate_holdout — following the route cancer_types and assays already take, so there is one way to pass a per-sample array, not two. Measured on the v0.0.32 stacking scores: AUC 0.8556, sample-level CI width 0.0129 against patient-clustered 0.0142 — a design effect of 1.21 and 10% wider intervals. That lands inside the reviewer's estimated 1.14-1.23 and agrees with the 1.26 measured independently on sens@98spec through a different code path. Existing CI numbers are therefore slightly too narrow, not wrong in direction; they will widen by about a tenth on the next run. Every result records `auc_<model>_ci_method`, and a missing PATIENT_ID logs `patient_column_missing` with its consequence spelled out. A silently narrow interval is the failure mode here, so the fallback announces itself rather than looking identical to the clustered path. Also logs ERR-20260824-001: the two local GrootCV test failures are arfs 2.4 against LightGBM 4.7 — the incompatibility the `arfs>=3.0.0` pin exists to prevent — reproduced on a clean develop checkout and therefore not a code bug.
feat(report,eval): intervals that account for clustering — on the primary endpoint, and everywhere else
Review item 1 asked whether the histology detection gap is label-tier composition rather than biology. Running it properly says: partly, and our first answer overstated what survives. The composition difference is real and large. In the 1-5% VAF band pancreatic samples are 47.5% True+ with a median of ZERO confirmed variants, against NSCLC's 75.7% and a median of two — which is exactly the incompatibility with the True+ definition that prompted the objection. Matched on confirmed-variant count within True+ only, one contrast is clean (NSCLC 41.2% vs bladder 20.3% at 4+ variants, OR 2.75, p=0.004, non-overlapping intervals), one is marginal (NSCLC 31.2% vs pancreatic 18.5% at 2-3, OR 2.01, p=0.049, intervals overlap), and one is null on 21 samples. Only the first survives a correction across the three. The pooled 36.1% vs 20.2% we sent the reviewer overstates it: the effect rests on the 4+ stratum. The full grid is deliberately not reported. Across 34 histologies and four strata, 14 of 136 cells reach 20 samples and the median cell holds one; a matrix of point estimates there would repeat the retinoblastoma odds-ratio error the review already caught. Cells below the floor print as UNTESTABLE, and the three contrasts are pre-specified in the script rather than chosen after looking.
research: the histology gap, matched on variant count within True+ only
…tier composition The subgroup panel printed retinoblastoma as "n=874, positives=72, AUC 0.701" in the same visual weight as NSCLC's "n=3707, positives=2254, AUC 0.793". An AUC on 72 positives is not comparable to one on 2,254, and the panel gave a reader no way to see that. It is the same denominator error that made a retinoblastoma odds ratio meaningless in review, in a different panel — and the report still carried it after we withdrew the odds ratio. Three changes, all so the reader can see the confound rather than being told a conclusion about it: The floor now counts the SCARCE class, not total n. A group could clear a 30-sample gate on its 874 negatives while carrying five positives. Each subgroup AUC carries a patient-clustered interval, reusing the eval engine's bootstrap so a subgroup interval and a headline interval mean the same thing. Retinoblastoma reads 0.701 [0.606, 0.780] against NSCLC's 0.793 [0.779, 0.809] — the width now says what the counts implied. Intervals are computed for the pre-registered primary evaluator only: the other 25 subgroup tables carry no declared claim, and bootstrapping all of them added ~100s to the report build for panels nobody has pre-registered a question about. Rows report the share of their positives that are tumour-confirmed. Melanoma is 51.4% True+ against bladder's 90.3%, so those two AUCs are discriminating quite differently certain sets of positives. What this deliberately does NOT do is import the C2 analysis. That finding — pancreatic positives being 47.5% True+ in the 1-5% VAF band — is band-specific; cohort-wide the spread narrows to 73-92%. The subgroup panel is not sliced by VAF, so putting the strong claim there would overstate it in exactly the way C2 caught us overstating the pooled contrast. The matched-stratum contrasts stay in the campaign record where they are pre-specified. Also fixes a pre-existing test that asserted an exact key set on subgroup rows, so it broke on any added aggregate; it now asserts a subset of allowed keys, which is what a no-identifiers guarantee actually needs.
Nine of the thirty items are closed and one was added on the way, so the register no longer described the work. Completed rows stay listed with their PR — what was tried and what it cost is the part that gets lost otherwise. Adds a findings section for what the batch established, because two of the three results change what we can claim rather than just closing items: the headline's uncertainty is dominated by threshold estimation (11.3x) rather than by the clustering everyone has been arguing about (1.26, measured twice through separate code paths and matching the reviewer's predicted range both times); and the histology gap narrows to the 4+ variant stratum, with the composition confound real but band-specific. Two decisions moved rather than resolved: shipping v0.0.33 now waits on B1 alone, and the reviewer response has gained enough measured content that holding it costs more than it did this morning. Records the two operational traps this batch hit — CIs from before #132 are not comparable, and CI does not run on stacked PRs.
feat(report): subgroup AUCs gain a per-class floor, an interval, and tier composition
docs: bring the register up to date after the local batch
Anchor choice moves the headline 13.1 AUC points — more than any modeling decision in the campaign — and the report stated that as the third clause of a grey caption under a table. The numbers were already computed; only their prominence was wrong. Four rungs now, scored on the same model and the same positives with only the negative class changing: verified within-patient negatives 0.845, all negatives pooled 0.856, unpaired negatives 0.865, healthy donors 0.976. The unpaired rung is new — it existed in the campaign record but never in the report — and every rung carries its own n, patient count, and a patient-clustered interval. The framing deliberately avoids a claim the data does not support. The donor rung is the thinnest at 55 negatives, and the tempting line is that the flattering number is also the unreliable one. Checked against two independent estimates (stratified two-sample bootstrap 0.025, Hanley-McNeil 0.014) our interval of 0.029 is the widest of the three, so the donor number is only ~1.7x less precise than the verified one. It is not untrustworthy; it answers a different question — between people rather than within a patient — and it is what most screening literature reports. That is what the panel says. A rung under 20 negatives is dropped rather than drawn with an interval its n cannot support, and the AUC-by-negative-class clause is removed from the operating-points note so one place owns the statement.
ERR-20260825-001. The docker (gpu) job failed on a pure-Python PR with GPGKeyTemporarilyNotFoundError from Launchpad while add-apt-repository fetched the deadsnakes signing key — an external 500, in the apt layer, before any kreview code is installed. The CPU image shares the builder stage and passed on the same commit, which is the cheapest way to tell this apart from a real break. Records the standing weakness too: the GPU stage sits on a CUDA image based on Ubuntu 22.04, whose distro python is 3.10, so every GPU build depends on a third-party PPA for 3.12. A 24.04 CUDA base ships 3.12 natively and would take that dependency out of the release container's build path.
feat(report): the verification-bias ladder as a panel, not a footnote
Version in the three places the release guide names (settings.ini, kreview/__init__.py, nextflow/nextflow.config) plus the changelog entry. Also repairs the release guide's own docs gate, which had been failing before this release and presumably before the last one: `mkdocs build --strict` aborted on seven griffe warnings — four parameters with no annotation (two duck-typed estimators, two array-likes) and `iter_feature_chunks`, a generator with no return annotation at all, so griffe could not match the three-tuple its docstring documents. Annotated rather than suppressed; the gate passes now.
Release-prep audit of README and docs against the code. Three classes of staleness, one of them a factual error in the files agents read on every turn. The taxonomy is SIX tiers, not five, and CH-only samples go to `Undetermined`. `.agents/rules/labeling-hierarchy.md` and the ctdna-labeling skill both said a CH-only Possible ctDNA+ is "demoted to Possible ctDNA−" — the code demotes it to `Undetermined` (labels.py, LABEL_UNDETERMINED), which is outside `_MODEL_LABELS` and therefore excluded from modelling entirely. The documented behaviour would have put CH-only samples in the negative class. docs/biology/ctdna-labeling.md contradicted itself: prose said Possible ctDNA−, its own code block three lines later said Undetermined. Eleven files carried "5-tier", including AGENTS.md, which is read on every turn; the labeler's own docstring has said 6-tier all along. The report interpretation guide described four tabs and a report that no longer exists. It now covers five, the primary endpoint and its interval, the verification-bias ladder (13.1 AUC points between anchors), the pipeline run map, the subgroup panel's per-class floor and tier composition, and why the interval width is mostly threshold estimation rather than clustering. Its broken `#page-5-cohort-qc` link — a leftover from the retired Quarto dashboard — now points at the section that replaced it. README also claimed Boruta-SHAP as a multimodal selection option two bullets above its own table naming grootcv the default; Boruta is now a legacy extra that cannot be installed alongside arfs at all. `mkdocs build --strict` passes clean, with no warnings and no broken links.
chore(release): bump version to 0.0.33
The API reference covered 5 of 13 modules. Absent were the report data layer, feature selection, the evaluation and selection CLIs, the registry internals, report rendering, reproducibility, and the pipeline diagram added last week — so the two modules that decide what a number in the report *means* (report_data) and which features reach a model (selection) were undocumented while the scoreboard that displays their output was not. Eight pages, each opening with what the module is responsible for rather than restating its name, then the mkdocstrings render. Nav ordered to follow the pipeline rather than the alphabet. `cli` stays on mkdocs-typer, which renders the real CLI help and is the right tool for a command surface. Two things this verified rather than assumed: no new griffe warnings appeared from eight previously undocumented modules, so their annotations were already sound; and the built pages carry real content (6, 5, 6 and 2 documented objects on the four spot-checked). `mkdocs build --strict` passes clean.
Step 4 said to land the version bump on develop and then PR develop into main. No release has done that. v0.0.32, v0.0.31 and v0.0.28 each merged their release/vX.Y.Z branch straight into main and were back-merged afterwards (#116/#117, #113/#114, #54) — git-flow, which is also what the release/<version> branch naming in git-conventions implies. The written path is not merely different, it is riskier: with develop as the staging area, anything a colleague merges there between the bump and the develop-to-main PR rides into the release unreviewed. On a repository taking several PRs a day that is a live hazard, and it nearly caught this release. Also records the v prefix that every existing release branch uses, states that the tag goes on main after the release PR merges, and makes the back-merge a PR to match #117/#114/#106 rather than a local merge and push.
docs: complete the API reference, and correct the release guide
The 0.0.33 entry was written before the release-prep docs audit and before #138, so it described the code changes and none of the documentation ones — including the correction that matters most to anyone reading the repository's rules: the taxonomy is six tiers, and CH-only samples are excluded from modelling rather than moved into the negative class, which is what the rules file and the labeling skill both said.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Release v0.0.33, following
docs/developer/release-guide.md— corrected in #138 to describe the git-flow this repo actually uses, so this branch targetsmainrather than routing through develop.60 commits since v0.0.32. The through-line is honest uncertainty: every headline number now carries an interval that accounts for how the cohort is actually structured, and the report states which negatives it was scored against.
What ships
The primary endpoint carries an interval. Sensitivity at 98% specificity against verified true negatives had shipped as a bare point estimate. It is now 0.483 [0.424, 0.519] from a patient-clustered bootstrap with the threshold re-estimated inside every resample. Decomposing that width separated two things three review rounds had conflated: clustering inflates the variance 1.26×, estimating the threshold 11.3×.
Every scoreboard interval was too narrow.⚠️ CI values from earlier releases are not comparable to these.
_bootstrap_aucresampled rows, so evaluator and holdout CIs treated a patient's repeated timepoints as independent. Design effect measured at 1.21 — intervals widen ~10%.The verification-bias ladder as a panel. Four anchors — verified TN 0.845, pooled 0.856, unpaired 0.865, donors 0.976 — spanning 13.1 AUC points, which is more than any modelling decision in the campaign.
Subgroup AUCs gain a per-class floor, patient-clustered intervals, and the share of each group's positives that are tumour-confirmed (51%–90% across histologies).
The Nextflow DAG drawn into the report, in Methods (structure) and Run diagnostics (this run's task counts), with a test asserting the declaration against the workflow in both directions.
Plus the report's visual rebuild, dual-anchor operating points, per-sample depth metadata, repeatability guards, and a completed API reference.
Release QA
nbdev-export+nbdev-cleanruff checkblack --checkmypy kreviewpytest.[arfs])mkdocs build --strictsettings.ini,kreview/__init__.py,nextflow/nextflow.configAfter this merges: tag
v0.0.33onmain(triggers PyPI, GHCR and versioned docs), verify the three artifacts, then back-mergemain→developby PR.