Proposal: out-of-distribution / extrapolation benchmark for inorganic solids (DFT-anchored)

[Gazzara-lab]

Hi! Some honesty up front: I'm new to open-source contribution — this would be my first. I've been building a small personal setup to help me scout research opportunities, and while it was going through mlipaudit it flagged something I then looked into more closely myself. Here's what I found, in case it's useful.

First, thanks for the tool — the breadth of property checks across small molecules, liquids and biomolecules is genuinely useful for deciding whether to trust a potential.

The gap I noticed: there doesn't seem to be a benchmark for out-of-distribution / extrapolation behavior on inorganic solids — arguably the most consequential (and often silent) MLIP failure mode. The current general benchmarks (Stability, Scaling) probe robustness and cost rather than how accuracy degrades once a model is pushed outside its training regime.

I'd like to contribute one. A first version would score a model on relative phase energetics against DFT ground truth — e.g. polymorph energy ordering (Kendall-τ and ground-state hit-rate) — and report an "OOD gap": the accuracy drop between an in-distribution control set and an out-of-distribution frontier set, evaluated for the same model. Scoring against DFT rather than inter-model consensus is deliberate, to avoid consensus/floor-saturation artifacts.

I'd implement it following the "Adding a new benchmark" tutorial as a new OOD category, starting small — polymorph ordering for a handful of compositions — as a single complete, tested PR (class + tests + UI + docs + a validated reference dataset on Hugging Face), and add further axes (pressure-driven phase stability, held-out chemistry) as follow-up PRs.

Since this would be my first contribution here, I'd really appreciate a sanity check before I invest the work: is a benchmark like this within scope, and would a PR be welcome? Happy to align on the scoring design first. Thanks for taking a look!

[lwalew]

Hi @Gazzara-lab , many thanks for opening this issue! We certainly welcome community contributions and would be glad to work with you on this. We're currently reviewing our guidelines for accepting new benchmarks from the community and will get back to you very soon with an update.

In the meantime, do you already have a specific dataset or DFT source in mind, and at what level of theory?

[Gazzara-lab]

Thanks @lwalew, really appreciate the warm response and the openness to community benchmarks.

Yes — here's what I have in mind, and I'd value your steer on it:

Level of theory. PBE (GGA/GGA+U, Materials Project / MP2020 settings). I'd anchor to PBE deliberately, so the reference matches the training distribution of the foundation MLIPs this would target (MACE-MP-0, CHGNet, M3GNet, SevenNet — all trained on MP/MPtrj). The point is for the "OOD gap" to isolate extrapolation error rather than functional disagreement: against a different functional, a PBE-trained model would look bad just for the mismatch.

Data sources (no new DFT for v1). I'd reuse already-published, consistently-computed PBE data so v1 is fully reproducible:

• In-distribution control set: held-out polymorph families from the Materials Project (compositions with several known structures and a settled PBE energy ordering).
• OOD frontier set: the WBM dataset (the Matbench-Discovery test set). It's already PBE and was generated by chemical-similarity elemental substitution of MP structures across 5 successive batches — so it comes with a built-in "distance from MP" gradient that's a natural OOD axis, no new calculations needed.

On cross-model fairness. Because v1 scores relative energetics (Kendall-τ on polymorph ordering + ground-state hit-rate) rather than absolute energies, it sidesteps the MP2020-corrected vs raw-energy differences between models (e.g. CHGNet uses MP2020-corrected energies, MACE-MP trains on raw MPtrj energies) — ordering is invariant to a per-composition shift, so comparisons stay apples-to-apples.

Optional follow-up. I could add an r2SCAN track later via MatPES (it ships both PBE and r2SCAN) to cover meta-GGA-trained models, keeping PBE as the v1 primary.

Does PBE-primary with a WBM-based frontier fit the guidelines you're putting together? Happy to swap the frontier source (Alexandria / GNoME are alternatives) or fold in r2SCAN from the start if you'd prefer. I'll align on the final scoring design with you before opening the PR.