Build against RDKit 2026.03, harden the C++ core, modernize the Python API

.gitignore

@@ -32,6 +32,7 @@ htmlcov/
 # Build artifacts
 lib/
 temp.*/
+*.log
 
-# PR documentation (not included in PR)
-PR_SPEEDUP_*.md
+# Local build env scratch
+validate_fix.py

BUILD.md

@@ -0,0 +1,78 @@
+# Building MolFTP
+
+MolFTP has a C++ core (`src/molftp_core.cpp`) with pybind11 bindings that links against
+**RDKit's C++ headers and libraries**. The standard `pip install rdkit` wheel is
+*runtime-only* — it does **not** ship the C++ headers — so it cannot build this extension.
+conda-forge's `rdkit` does ship them (and pulls in Boost), so that is the supported path.
+
+## Quick start (recommended)
+
+```bash
+# 1. Create the environment — RDKit 2026.03 + build deps, one command
+conda env create -f environment.yml      # or: mamba env create -f environment.yml
+conda activate molftp
+
+# 2. Build + install in editable mode
+pip install -e .
+
+# 3. Verify
+python -c "import molftp; print('molftp', molftp.__version__, 'OK')"
+pytest -q                                 # optional: run the test suite
+```
+
+`setup.py` auto-detects RDKit from the active conda env via `$CONDA_PREFIX` — no paths to
+edit, no environment variables to set.
+
+## How detection works
+
+`setup.py` resolves the RDKit prefix in this order:
+
+1. **`RDKIT_PREFIX`** — explicit override (set this to use a custom RDKit build).
+2. **`CONDA_PREFIX`** — the active conda env (the recommended path above).
+
+It then adds the right include directories (handling both the conda-forge
+`include/rdkit/GraphMol/...` layout and the plain `include/GraphMol/...` layout) and links
+the seven RDKit libraries the core needs. An `-rpath` to the env's `lib/` is baked in, so
+the RDKit dylibs are found at import time **without** any `DYLD_LIBRARY_PATH` /
+`LD_LIBRARY_PATH` juggling.
+
+## Custom RDKit location
+
+If you built RDKit yourself (headers under `<prefix>/include/rdkit/` and libs under
+`<prefix>/lib/`):
+
+```bash
+export RDKIT_PREFIX=/path/to/your/rdkit/prefix
+pip install -e .
+```
+
+## Requirements
+
+- A **C++20** compiler (clang on macOS, gcc or clang on Linux). RDKit 2026.03's headers use
+  C++20 features (`constexpr virtual`, `constexpr` destructors), so C++17 no longer compiles.
+  `setup.py` sets `cxx_std=20`.
+- RDKit **2026.03** is what we build against; 2022.03+ is expected to work.
+- Tested on macOS (Apple Silicon) and Linux x86-64.
+
+## Why conda-forge (the dev-package split)
+
+conda-forge splits RDKit into separate packages, and this trips up most build attempts:
+
+| package | ships | needed to… |
+|---|---|---|
+| `rdkit` | Python module + runtime libs | *run* molftp |
+| `librdkit-dev` | **C++ headers** (`include/rdkit/...`) + dev symlinks | *build* molftp |
+| `libboost-devel` | **Boost headers** (RDKit headers `#include <boost/...>`) | *build* molftp |
+
+`environment.yml` lists all three. Installing only `rdkit` is the #1 cause of
+"RDKit C++ headers not found" and "`boost/...` file not found".
+
+## Troubleshooting
+
+| Symptom | Cause / fix |
+|---|---|
+| `ERROR: RDKit C++ headers not found` at build | Missing `librdkit-dev` (or not in the conda env / used the pip `rdkit` wheel). Re-create from `environment.yml`, `conda activate molftp`, or set `RDKIT_PREFIX`. |
+| `fatal error: 'boost/...' file not found` | Missing `libboost-devel`. Re-create the env from `environment.yml`. |
+| `error: constexpr ... virtual function cannot be constexpr` (in `Geometry/point.h`) | Compiling RDKit 2026.03 headers as C++17. Ensure `cxx_std=20` (already set) and a C++20-capable compiler (`cxx-compiler` from conda-forge). |
+| `ImportError: library not loaded ... libRDKit*.dylib` at import | RDKit dylibs not on the loader path. Import from the same conda env you built in; the baked-in `-rpath` handles this automatically. |
+| Linker can't find `-lRDKit*` | RDKit libs missing from the env. Re-create from `environment.yml`. |

README.md

@@ -2,7 +2,8 @@
 
 [![License: BSD-3-Clause](https://img.shields.io/badge/License-BSD%203--Clause-blue.svg)](https://opensource.org/licenses/BSD-3-Clause)
 [![Python 3.8+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
-[![C++17](https://img.shields.io/badge/C++-17-blue.svg)](https://isocpp.org/)
+[![C++20](https://img.shields.io/badge/C++-20-blue.svg)](https://isocpp.org/)
+[![RDKit 2026.03](https://img.shields.io/badge/RDKit-2026.03-green.svg)](https://www.rdkit.org/)
 
 High-performance molecular feature generation based on fragment-target prevalence statistics. MolFTP generates interpretable, statistically-grounded features for molecular property prediction with state-of-the-art performance.
 
@@ -26,37 +27,34 @@ High-performance molecular feature generation based on fragment-target prevalenc
 
 ### Requirements
 
-- Python >= 3.11
-- RDKit >= 2025.3.0
-- NumPy >= 1.19.0
-- C++17 compatible compiler
+- Python >= 3.9
+- RDKit (latest tested: **2026.03**; 2022.03+ expected to work)
+- A **C++20** compiler (clang on macOS, gcc/clang on Linux) — RDKit 2026.03 headers use C++20
+- NumPy, pandas, scikit-learn
 
-### Install from source
+MolFTP has a C++ core that links against RDKit's **C++ headers and libraries**. The plain
+`pip install rdkit` wheel is runtime-only and cannot build it — you need the conda-forge dev
+packages (`librdkit-dev` + `libboost-devel`). `environment.yml` sets all of this up in one step.
+
+### Install from source (recommended)
 
 ```bash
-# Clone the repository
 git clone https://github.com/osmoai/molftp.git
 cd molftp
 
-# Create and activate conda environment with build tools
-mamba create -n rdkit_dev cmake librdkit-dev eigen libboost-devel compilers
-conda activate rdkit_dev
+# One command — RDKit 2026.03 + C++ headers (librdkit-dev) + Boost (libboost-devel) + toolchain
+conda env create -f environment.yml      # or: mamba env create -f environment.yml
+conda activate molftp
 
-# Install Python dependencies
-conda install -c conda-forge numpy pandas scikit-learn
-conda install -c conda-forge rdkit
+# Build + install (editable). setup.py auto-detects RDKit from the active env.
+pip install -e .
 
-# Build and install
-python setup.py install
+# Verify
+python -c "import molftp; print('molftp', molftp.__version__, 'OK')"
 ```
 
-**Note**: Use `mamba` for faster dependency resolution, or replace with `conda` if mamba is not installed.
-
-### Quick install with pip (coming soon)
-
-```bash
-pip install molftp
-```
+See **[BUILD.md](BUILD.md)** for build internals, a custom-RDKit (`RDKIT_PREFIX`) path, and
+troubleshooting.
 
 ## Quick Start
 
@@ -98,6 +96,23 @@ print(f"Multi-task features shape: {features.shape}")
 # Features shape: (3, 81)  # 27 features per task × 3 tasks
 ```
 
+### End-to-end prediction (`SMILES → label`)
+
+The API is split into an **inference** layer (features) and a **predict** layer (labels):
+
+```python
+from molftp.predict import MolFTPClassifier
+
+clf = MolFTPClassifier(radius=6, method='key_loo', k_threshold=2).fit(train_smiles, y_train)
+labels = clf.predict(test_smiles)
+proba  = clf.predict_proba(test_smiles)[:, 1]
+X      = clf.transform(test_smiles)     # inference only: features, no prediction
+```
+
+`MolFTPClassifier` composes a molFTP feature generator with any scikit-learn estimator
+(`estimator=`, default `LogisticRegression`). See **[docs/api.md](docs/api.md)** for the full API,
+the inference/predict separation, parameter semantics (incl. `k_threshold`), and method notes.
+
 ## Examples
 
 See the `examples/` directory for comprehensive examples:
@@ -111,8 +126,8 @@ See the `examples/` directory for comprehensive examples:
 
 ### Key-LOO (Key Leave-One-Out)
 
-- Filters keys appearing in <= k molecules (default k=2)
-- Applies rescaling factor: `(n - k) / n` for better extrapolation
+- Filters rare keys: keeps a key only if its per-molecule **and** total counts are `>= k_threshold` (default 2)
+- `k_threshold` is passed through to the C++ core and genuinely changes the features (see [docs/api.md](docs/api.md))
 - Best for: Final model training, prediction on new molecules
 - Features are **task-independent** (can be pre-computed once)
 

docs/api.md

@@ -0,0 +1,148 @@
+# MolFTP API
+
+MolFTP turns molecules (SMILES) into interpretable, statistically-grounded feature vectors
+based on **fragment-target prevalence**, and (optionally) predicts labels from them. The Python
+API is split into two clearly separated layers:
+
+| layer | module | does | entry points |
+|---|---|---|---|
+| **inference** | `molftp.features` (a.k.a. `molftp.prevalence`) | SMILES → feature vectors | `MultiTaskPrevalenceGenerator`, `PrevalenceGenerator` |
+| **predict** | `molftp.predict` | SMILES → labels / probabilities | `MolFTPClassifier` |
+
+```
+inference:  SMILES ──fit/transform──▶ feature vectors
+predict:    SMILES ──features──estimator──▶ labels / probabilities
+```
+
+The predict layer never re-implements feature generation — it *composes* a feature generator
+with a scikit-learn estimator. Use the inference layer alone when you want features for your own
+model; use the predict layer for an end-to-end `SMILES → label` estimator.
+
+---
+
+## Install / build
+
+See [BUILD.md](../BUILD.md). In short:
+
+```bash
+conda env create -f environment.yml && conda activate molftp
+pip install -e .
+```
+
+---
+
+## Inference layer — feature generation
+
+### `MultiTaskPrevalenceGenerator` (recommended)
+
+Multi-task generator with NaN-sparse label support, backed by the C++ core.
+
+```python
+import numpy as np
+from molftp.features import MultiTaskPrevalenceGenerator
+
+gen = MultiTaskPrevalenceGenerator(radius=6, method="key_loo", k_threshold=2)
+gen.fit(train_smiles, y_train)            # y_train: (n,) or (n, n_tasks) with NaN for missing
+X_test = gen.transform(test_smiles)       # -> np.ndarray, shape (n_test, n_tasks * 3*(2+radius+1))
+```
+
+Feature width per task is `3 * (2 + radius + 1)` — three views (1D single fragments, 2D pairs,
+3D triplets), each contributing `2 + radius + 1` aggregated statistics. For `radius=6` that is
+27 features per task.
+
+Persistence (uses the C++ `py::pickle` protocol under the hood):
+
+```python
+gen.save_features("model.pkl")
+gen2 = MultiTaskPrevalenceGenerator.load_features("model.pkl")
+# or plain pickle — the generator round-trips correctly:
+import pickle; gen2 = pickle.loads(pickle.dumps(gen))
+```
+
+### `PrevalenceGenerator`
+
+Single-task generator (`fit` / `transform` / `fit_transform`). Returns the three per-view
+matrices; concatenate for a flat feature matrix.
+
+---
+
+## Predict layer — `MolFTPClassifier`
+
+A scikit-learn-style classifier: `fit` / `predict` / `predict_proba` / `transform`.
+
+```python
+from molftp.predict import MolFTPClassifier
+
+clf = MolFTPClassifier(radius=6, method="key_loo", k_threshold=2).fit(train_smiles, y_train)
+labels = clf.predict(test_smiles)
+proba  = clf.predict_proba(test_smiles)[:, 1]
+X      = clf.transform(test_smiles)        # inference only (features), no prediction
+```
+
+- `estimator=` — supply any scikit-learn estimator (default `LogisticRegression(max_iter=1000)`).
+  `predict_proba` requires an estimator that implements it.
+- `generator=` — supply a pre-configured `MultiTaskPrevalenceGenerator` instead of the keyword args.
+- For leakage-safe cross-validation, fit a fresh classifier **per fold** on that fold's training
+  molecules only.
+
+---
+
+## Methods: `key_loo` vs `dummy_masking`
+
+- **`key_loo`** — Key Leave-One-Out. Counts key occurrences, **filters rare keys**
+  (`k_threshold`, see below), and applies a Key-LOO rescaling pass on training rows. Use when you
+  fit on the full (train+valid) set and want rare-fragment filtering.
+- **`dummy_masking`** — builds full prevalence without rare-key filtering; at inference,
+  out-of-sample molecules use the **frozen** fitted prevalence (keys unseen in training contribute
+  0). Call `transform(smiles)` with **no** `train_indices_per_task` for out-of-sample inference.
+
+---
+
+## Parameter reference
+
+| parameter | default | meaning |
+|---|---|---|
+| `radius` | 6 | Morgan radius for fragment enumeration. |
+| `method` | `key_loo` | `key_loo` or `dummy_masking` (see above). |
+| `k_threshold` | 2 | **Key-LOO rare-key filter.** A key is kept only if its per-molecule count *and* its total count are `>= k_threshold`. `1` keeps everything; `2` drops keys seen in a single molecule; `3` drops keys seen in ≤2. Higher = more aggressive filtering of rare fragments. |
+| `nBits` | 2048 | Fingerprint width for the similarity/pairing step. |
+| `sim_thresh` | 0.5 | Tanimoto threshold for forming 2D/3D fragment pairs/triplets. |
+| `stat_1d` / `stat_2d` / `stat_3d` | `chi2` / `mcnemar_midp` / `exact_binom` | Significance test per view. |
+| `alpha` | 0.5 | Additive smoothing on contingency cells. |
+| `num_threads` | -1 | `-1` = all cores, `0` = auto, `>0` = fixed. |
+| `margin_mode` | `signcount` | Aggregation for the per-view `V[0]/V[1]` margin features. `signcount` (default, back-compat): net `(pos − neg)` atom count. `magnitude` (paper eq. 5): `max(+) − min(−)` of the atom-localized scores — small, consistent accuracy edge for LR/RF (see [research-notes.md](research-notes.md)). `both`: concatenate the two (`+2` features per view). |
+
+### Notes on `k_threshold`
+
+`k_threshold` is passed all the way through to the C++ core and genuinely changes which keys
+survive (and therefore the features). Earlier releases stored it in Python but did **not** pass it
+to C++ (it was hardcoded to 2); that is fixed — `test_regressions.py::test_k_threshold_changes_features`
+guards against a regression.
+
+### Sign-count features: why the LOO magnitude rescale and `loo_smoothing_tau` are inert
+
+molFTP's three-view features are **sign-based net counts**: for each view, `transform` counts the
+atoms whose aggregated fragment prevalence is ≥ 0 (PASS-leaning) vs ≤ 0 (FAIL-leaning) and reports
+the net `(pos − neg)` (overall and per depth). Only the **sign** of each prevalence value matters,
+never its magnitude. Two consequences worth knowing:
+
+- The Key-LOO `(k_j−1)/k_j` prevalence rescale and `loo_smoothing_tau`'s `(k_j−1+τ)/(k_j+τ)` are
+  **positive scalars** — they preserve every sign, so they **cannot change the features**. This is a
+  property of the feature design, not a no-op bug. Passing a training mask therefore yields the same
+  features as plain inference (pinned by
+  `test_kloo_core.py::test_positive_rescale_is_inert_for_sign_count_features`). `loo_smoothing_tau`
+  is additionally not wired into the C++ core; setting it ≠ 1.0 emits a `RuntimeWarning`.
+- The **effective** rare-key / leakage control is **`k_threshold`**, which *removes* keys (singletons
+  at `k_threshold ≥ 2`) and so does change the counts — see
+  `test_regressions.py::test_k_threshold_changes_features`. A label-permutation test confirms the
+  pipeline is leakage-safe in practice.
+
+If you want the LOO magnitude correction to actually influence the model, the aggregation in
+`build_3view_vectors_batch` would need to be made magnitude-aware (e.g. summing/maxing *signed*
+prevalence instead of counting signs) — a deliberate change that would shift all downstream results.
+
+> **Note (paper ↔ code):** the paper (arXiv:2510.06029, eq. 5) defines the *margin* feature as
+> `max(positive) − min(negative)` — magnitude-based — whereas the default `"max"` code path computes
+> a sign-count for it. The paper's own Figure 6 confirms the rescale is inert on the proportion
+> features. See [research-notes.md](research-notes.md) for the full analysis and the (a/b/c) decision,
+> which needs re-benchmarking before adopting the magnitude margin.

docs/dev-notes/README.md

@@ -0,0 +1,8 @@
+# Archived development notes
+
+Historical, point-in-time development artifacts (PR bodies, phase summaries, changelogs,
+build/test logs) from earlier MolFTP work. Kept for provenance only — **not** maintained
+and not part of the public docs. They were moved here from the repository root to declutter it.
+
+For current build instructions see [`../../BUILD.md`](../../BUILD.md); for usage see the
+top-level [`../../README.md`](../../README.md).