Add Ti64 and diamond phonon benchmarks#391
Conversation
ElliottKasoar
added
the
new benchmark
|
Hey @7radians, have you had a look at adding the thermal conductivity and Grüneisen parameter? would be good to get those in! For the calc, analysis etc, i think we will have a look at writing some more general stuff that you can use, as many people want to look at phonons in some shape or form. will update you on the progress of that |
|
Hey @7radians ive now made a PR for the phonon calc in a simpler way than what i showed you originally. it would be great to match this, but for your application we won't need the multiple workers. let me know if you need any help! |
…o phonons_Ti_C
# Conflicts: # docs/source/user_guide/benchmarks/bulk_crystal.rst
|
Hey @joehart2001, all done, phonons refactored and Gruneisen + thermal conductivity added. |
ElliottKasoar
left a comment
ElliottKasoar
left a comment
There was a problem hiding this comment.
Hi @7radians, thanks for this! Apologies for such a delayed review. I'm still taking a look at some parts, but I thought there's probably enough to start the discussion.
In addition to the above comments, please can you take a look at our new filtering guidelines: https://ddmms.github.io/ml-peg/developer_guide/filter.html.
The principles are relatively simple, but you do have to be a little careful, so again, if anything is unclear, please do ask! The biggest thing to check is that the info.json file is created during analysis, and this is read by the app.
(You'll need to rebase/merge the latest main for this)
One other question was whether it might make more sense to combine these into a single test? @joehart2001 may also have thoughts, but I think I recall some uncertainty about whether it's appropriate to have the diamond phonons as a standalone test compared to the full set of phonons in the current benchmark we have, for example.
There was a problem hiding this comment.
I think this is reverting us to an older form of the file?
There was a problem hiding this comment.
Sorry, accidental revert from a stale local state. Restored now, the file is identical to main
| GITHUB_BASE = "https://raw.githubusercontent.com/7radians/ml-peg-data/main" | ||
|
|
||
| EXTRACTED_ROOT = Path( | ||
| download_github_data( | ||
| filename="diamond_data/data.zip", | ||
| github_uri=GITHUB_BASE, | ||
| ) | ||
| ) |
There was a problem hiding this comment.
Are you happy for these to stay here, or would you prefer them to be in our S3 bucket?
There was a problem hiding this comment.
I'm happy either way, what would you prefer?
There was a problem hiding this comment.
for consistency we can do s3?
| Input structures (https://github.com/7radians/ml-peg-data/tree/main/diamond_data): | ||
|
|
||
| * ``diamond.yaml``: conventional cubic cell (8 atoms, a = 3.56 Å) × 4×4×4 = 512-atom | ||
| supercell, used for all MLIP band and thermal calculations. |
There was a problem hiding this comment.
Are these structures from the reference? It's worth clarifying, and giving more explanation if not.
There was a problem hiding this comment.
diamond.yaml is gone, the structure comes from the reference .xyz
There was a problem hiding this comment.
Very general question, but are you aware of/have you thought much about selection of these good and bad thresholds?
If you're happy with, great, but if not, we're happy to discuss/clarify!
There was a problem hiding this comment.
Yes, I've calibrated these deliberately, partly against physical scales and partly by how strict I want the benchmark to be. Please let me know what you think about these, I'm happy to discuss!
Dispersion RMSE mean (bad = 2.0 THz, weight 2): 20% of the ~10 THz Ti64 spectrum, the scale at which soft-mode and phase-stability conclusions can become unreliable.
Dispersion RMSE max (bad = 2.0 THz, weight 0.5): intentionally the same absolute bar applied to the worst case, a strictness statement that no single phase may be qualitatively wrong, with a low weight so it acts as a tiebreaker rather than dominating.
ω_avg MAE (bad = 1.0 THz): 10% of ω_max; this isolates systematic softening/stiffening of the lattice (sign-cancelling shape errors are already captured by the RMSE, hence the separate metric).
ΔF at 0 K (bad = 0.007 eV/atom): physically anchored to two scales. It's 20% of the ~35 meV/atom zero-point energy, and it's the scale of free-energy differences that decide phase competition in Ti alloys (tens of meV/atom), so errors beyond this scramble phase ordering.
ΔF at 2000 K (bad = 0.03 eV/atom): the same phase-stability logic at high temperature. |F| itself grows to ~1 eV/atom, but the differences that matter for the α/β competition stay at tens of meV/atom, so the absolute bar stays tight. This is knowingly the strictest metric in the set, current foundation models sit at or beyond it, which I think is fair. High-temperature free energies are where all models are weakest.
There was a problem hiding this comment.
it could be good to commnet on this very briefly in the docs. also to define what terms such as "unreliable" mean, as these will need to be explained fully for your section of the paper, so probably easiest to do in one go
| OUT_PATH = CALCS_ROOT / "bulk_crystal" / "ti64_phonons" / "outputs" | ||
|
|
||
| FMAX = 0.001 | ||
| STEPS = 10000 |
There was a problem hiding this comment.
Do we actually want to optimise for 10,000 steps?
@joehart2001 may have more experience with this from the other phonon benchmark (where we use 1000, I believe), but from other benchmarks, my impression is the only models that keep going beyond a few hundred steps are ones that explode, in most cases, so this can slow things down enormously when we run it for less suited models.
There was a problem hiding this comment.
Reduced to 1000 + switched to FIRE, matching the general phonon benchmark (kept fmax=0.001 as the cells are small, happy to align that too if you prefer). I used the settings from the reference paper initially but this seems to work well
There was a problem hiding this comment.
Same as the other metrics: are you aware of/have you thought much about selection of these good and bad thresholds?
There was a problem hiding this comment.
Same here, a mix of physical anchors and deliberate strictness. Again, please let me know if you have any thoughts about this, I'm happy to discuss!
Band MAE (bad = 1.0 THz, weight 2): 1 THz is 2.5% of diamond's 40 THz spectrum. This is a strictness choice. diamond is a simple, rigid, well-sampled crystal, so I want only quantitatively accurate force constants to score here.
Band RMSE (bad = 2.0 THz): the outlier-sensitive counterpart, at 2× the MAE bar since RMSE runs systematically higher for the same data.
Δγ (bad = 0.2): physically anchored - γ enters thermal expansion linearly (α ∝ γ), so 0.2 on γ_ref ≈ 1.01 means thermal expansion wrong by 20%, and it's also roughly the spread among experimental determinations of diamond's γ, i.e. beyond it a model is distinguishably wrong.
Δθ_D (bad = 90 K): θ_D here is derived from ω_max, at ≈48 K per THz,so 90 K ≈ a 1.9 THz error in the stiffest phonon, deliberately consistent with the band-structure bars above rather than an independent scale.
Δκ_L (bad = 400 W/m/K): κ_Slack ∝ θ_D³/γ² amplifies the other two errors, a model right at the Δγ bar alone already produces a ~45% κ error, so 400 (≈53% of κ_ref = 751) is approximately the propagation of the γ and θ_D thresholds through the Slack formula, keeping the three thermal metrics mutually consistent.
| model = load_models(model_name)[model_name] | ||
| calc = model.get_calculator() | ||
|
|
||
| for spec in CASES: |
There was a problem hiding this comment.
It might be nice to show process over this loop?
There was a problem hiding this comment.
Can you set a (default) spin (multiplicity) and charge for the structures, as this is required by some models.
There was a problem hiding this comment.
At the point when we run the app, we should generally just be loading figures etc. I think a lot of this should probably be moved to analysis?
Have you also checked if similar things are already done for the general phonon benchmark?
There was a problem hiding this comment.
Both modules are deleted. Ti64 reuses the general benchmark's render_dispersion_component/lookup_system_entry directly (the shared formats made that possible). For diamond, the dispersion comparison is pre-rendered as a PNG during analysis and the app just loads it, so no shared-module changes needed for either.
There was a problem hiding this comment.
As with the other app, I think a lot of this ought to be done in analysis, and I'd also check for overlap with existing functions.
There was a problem hiding this comment.
Removed this module as mentioned above
| Computational cost | ||
| ------------------ | ||
|
|
||
| Medium: band-structure and thermal-property calculations together typically take |
There was a problem hiding this comment.
| Medium: band-structure and thermal-property calculations together typically take | |
| Low: band-structure and thermal-property calculations together typically take |
Compared to other benchmarks (and given we're not marking the test as @slow / very_slow
| Computational cost | ||
| ------------------ | ||
|
|
||
| Medium: dispersion, DOS/PDOS and thermodynamic calculations typically take minutes per model on CPU. |
There was a problem hiding this comment.
| Medium: dispersion, DOS/PDOS and thermodynamic calculations typically take minutes per model on CPU. | |
| Low: dispersion, DOS/PDOS and thermodynamic calculations typically take minutes per model on CPU. |
Compared to other benchmarks (and given we're not marking the test as @slow / very_slow
…enchmarks - Restore models.py to main's version (accidental revert) - Make is_plusminus an explicit parameter of init_phonopy_from_ref so the general phonon benchmark's behaviour is unchanged - Use ase.units constants in thermal_utils; add shared harmonic_free_energy and qpath_distances helpers - Rewrite diamond and Ti64 calcs to write the general phonon benchmark's output formats (band/DOS pickles, thermal JSON, relaxed .xyz), with DFT references converted under outputs/DFT - Relax with FIRE + FixSymmetry (1000 steps) as in the general benchmark; explicit get_calculator(precision="high"); default charge/spin; catch per-case errors and show progress (filtering guidelines) - Analyses write info.json for element filtering and reference calc-stage data paths; apps only load pre-computed data (no downloads at app time) - Delete both custom app helper modules; reuse the general phonon dispersion/DOS renderer (now tolerates missing DOS) - Docs: cost wording (Low), structure provenance, updated method details
- CASTEP reader: remove global warnings suppression and dead xscale rescaling; keep only frequency/q-point parsing and segment splitting - Ti64: strict single-pass weight parser that raises on count mismatch; idempotent ASE reader patch; minimal per-case specs with k-path derived from the lattice prefix; skip completed models before loading calculators - Diamond: remove unused 6x6x6 mesh evaluation; cache reference band parsing - thermal_utils: share gaussian_dos and EV_TO_KJMOL; reuse the Gruneisen mesh frequencies for the Debye temperature instead of a second mesh run - Ti64 analysis: discover cases from reference outputs instead of importing the calc module; use the shared rmse helper - Diamond analysis: store scatter asset paths once per model, not per point - Apps: fix Sphinx docs anchors; derive fallback columns from table metrics - Add fast unit tests for the shared phonon helpers
- Diamond/Ti64 docs: references are independent of public DFT databases; chemistry is in-domain for Alexandria-derived training sets (OMat24), but the Ti64 substituted supercells do not appear in them verbatim - Use np.trapezoid in tests (np.trapz removed in numpy 2.x)
The ti64_data zip now ships the CASTEP (PBE) references pre-converted to the shared phonon benchmark formats, so the reference step is a plain copy and the CASTEP log parser, q-point weight parsing, and the ASE reader energy patch are no longer needed. Model relaxations start from the reference structures directly.
The diamond_data zip will ship the CASTEP/RSCAN reference pre-converted to the shared phonon benchmark formats plus q-path metadata, so the reference step becomes a plain copy and the model band path is read from diamond_qpath_metadata.pkl. Remove the now-unused conversion helpers (qpath_distances, gaussian_dos, harmonic_free_energy) and their tests.
The diamond reference data now ships fully pre-converted, and the MLIP calculations regenerate the canonical +-0.01 A displacement pair with phonopy (verified identical to the reference dataset), so diamond.yaml is no longer needed. Structures come from diamond.xyz and the band path from the q-path metadata.
Add load_pickle/load_json to analysis utils and use them in the general, diamond, and Ti64 phonon analyses instead of per-benchmark loaders. Revert the app utils typing shim to main's version (no longer needed on Python 3.12) and drop the phonon helper unit tests.
With the pre-converted reference data, both steps use identical phonopy setups (same relaxed cell, 4x4x4 supercell, +-0.01 A displacements), so compute the equilibrium force constants once. Removes two of the six 512-atom force evaluations per model.
PhonopyGruneisen's gamma formula divides by the full strain span between the plus and minus cells; passing our half-span delta_strain doubled every mode gamma and quartered the Slack conductivity. Let phonopy derive the span from the actual cell volumes instead (verified: old/new ratio exactly 2.0 on a Lennard-Jones solid, corrected diamond reference gamma 1.01 matches literature). Rescale the Dgamma/Dkappa thresholds to the corrected scale, remove the now-unused Debye helper, and drop the unused n_atoms_primitive parameter.
The CASTEP reference stored branches in mode-continuity order rather than frequency order; the reference data now ships frequency-sorted per q-point, matching the phonopy convention used by all model outputs, so the analysis no longer needs its own branch sorting.
Restore interactive_helpers, analyse_phonons, and analysis utils to main. The diamond dispersion preview is pre-rendered during analysis and served via the existing image fallback in the shared renderer, and the diamond/Ti64 analyses carry their own small file loaders.
Points carry unique diamond-{i} ids again (as in the original
implementation) so hover and click selection behave per point, and the
per-model dispersion image and structure paths move to the model entry,
with the app resolving any click to the model's pre-rendered preview.
Diamond: band MAE bad 1.0 THz with weight 2, thermal metrics at Dgamma 0.2, Dtheta_D 90 K, Dkappa 400 W/m/K, all weight 1. Ti64: dispersion RMSE mean weighted 2, max RMSE bad 2.0 THz, omega_avg MAE bad 0.5 THz.
|
Hi @ElliottKasoar, thank you very much for the review! All points should now be addressed, I've left replies on the individual threads, but the short version: I've pre-converted the reference data to the general phonon benchmark's formats (band/DOS pickles, thermal JSON, structures), and this simplified things a lot and allowed sharing functions throughout: both calcs now use the general benchmark's init_phonopy_from_ref/get_fc2_and_freqs, write the same per-system output files with references under outputs/DFT/, and the apps reuse the general dispersion renderer, so the custom CASTEP parser, the ASE reader patch, diamond.yaml, and both app helper modules are gone entirely, and the reference steps are now just a copy of the downloaded data. Outside the two benchmark folders, the only changes left are the is_plusminus parameter in phonons_utils (as you suggested, defaulting to the current behaviour) and the new thermal_utils module. I've also merged the latest main and followed the filtering guidelines (info.json from analysis, info_path in the apps, calcs catch per-case errors and run with the mock calculators), and while going through thermal_utils I found and fixed a factor-of-2 error in the Grüneisen parameter, the corrected diamond reference gives γ = 1.01, in line with literature. All model results have been regenerated with the corrected pipeline. Happy to move the data zips to the S3 bucket, and to discuss the metric thresholds, some I've recalibrated with physical scales in mind (please see the replies in the thread), and some others I guess are just about how strict we want to be. |
Compute the model total DOS with the shared phonopy helpers (run_mesh + run_total_dos, Ti64 pattern), add diamond_dos.npz to the DFT reference files, and render band + DOS on the fly in the app via the shared phonon helpers instead of pre-rendered dispersion PNGs. Document the DOS in the diamond benchmark docs.
3 participants
Pre-review checklist for PR author
Summary
This PR adds two new bulk-crystal phonon benchmarks with Dash app integration:
Ti-6Al-4V (Ti64) phonons (
bulk_crystal/ti64_phonons)Raw outputs are written to
outputs/<model>/as<case>.npz+<case>.json.metrics.jsonplus:ti64_phonons_metrics_table.json(Dash table)ti64_phonons_interactive.json(interactive scatter)Metrics include dispersion RMSE (mean/max), ω_avg MAE, and ΔF per-atom at 0 K / 2000 K for TP-enabled cases.
Diamond phonons (bands-only) (
bulk_crystal/diamond_phonons)FORCE_CONSTANTS+band.yamlunderoutputs/<model>/. The phonon band path is taken directly from the DFT reference NPZ (dft_band.npz) so predictions match the exact reference q-path.band.yaml, sorts bands per q-point and reports Band MAE + Band RMSE (THz). Writes:diamond_phonons_bands_table.jsondiamond_phonons_bands_interactive.jsonband.yamlwith optional DFT overlay (RSCAN).Linked issue
Resolves #297
Progress
Testing
Tested both benchmarks on:
New decorators/callbacks