Skip to content

Added benchmark CRBH20 reaction energy barriers dataset#343

Merged
ElliottKasoar merged 17 commits into
ddmms:mainfrom
Quantumplations:main
Jul 10, 2026
Merged

Added benchmark CRBH20 reaction energy barriers dataset#343
ElliottKasoar merged 17 commits into
ddmms:mainfrom
Quantumplations:main

Conversation

@Quantumplations

Copy link
Copy Markdown
Contributor

Pre-review checklist for PR author

PR author must check the checkboxes below when creating the PR.

Summary

  • Input data: CRBH20 (Chemical Reaction Barrier Heights 20), a set of 20 molecular geometries of reactants and their transition states (cycloreversion of heterocyclic rings).
  • Calculations: predicted reactant and transition state energies using 6 foundation models (mace-mp-0a, mace-mp-0b3, mace-mpa-0, mace-omat-0, mace-matpes-r2scan, orb-v3-consv-inf-omat, pet-mad), along with the reaction barrier height (difference between transition state and reactant energies).
  • Analysis: compare the 6 models' predicted reaction barrier heights against r2SCAN DFT reference barrier heights, metric for each model is its MAE over the 20 systems.

Linked issue

Resolves 1 issue.

Resolves #291

Progress

  • Calculations
  • Analysis
  • Application
  • Documentation

Testing

  • mace-mp-0a, mace-mp-0b3, mace-mpa-0, mace-omat-0, mace-matpes-r2scan, orb-v3-consv-inf-omat, pet-mad

New decorators/callbacks

@ElliottKasoar ElliottKasoar added the new benchmark Proposals and suggestions for new benchmarks label Feb 6, 2026
@Quantumplations

Copy link
Copy Markdown
Contributor Author

Hey sorry for the long delay in getting on top of this! I've rebased the PR, changed the naming to CRBH20, did the pre-commit formatting, and also I removed some raw data from the repo. I'm also attaching the data zip file for the S3 bucket. Please review and let me know if there's anything else that needs to be done!
CRBH20.zip

Comment thread ml_peg/calcs/molecular/CRBH20/calc_CRBH20.py Outdated
Comment thread ml_peg/calcs/molecular/CRBH20/calc_CRBH20.py Outdated
Comment thread ml_peg/calcs/molecular/CRBH20/calc_CRBH20.py Outdated
Comment thread ml_peg/calcs/molecular/CRBH20/calc_CRBH20.py Outdated
Comment thread ml_peg/calcs/molecular/CRBH20/calc_CRBH20.py Outdated
Comment thread ml_peg/calcs/molecular/CRBH20/calc_CRBH20.py Outdated
@joehart2001

Copy link
Copy Markdown
Collaborator

Hey sorry for the long delay in getting on top of this! I've rebased the PR, changed the naming to CRBH20, did the pre-commit formatting, and also I removed some raw data from the repo. I'm also attaching the data zip file for the S3 bucket. Please review and let me know if there's anything else that needs to be done! CRBH20.zip

I've uplaoded and written a code suggestion for the download. If you accept hte code suggestion, remember to fetch and pull the changes to your local machine before doing any edits, otherwise they will be overwritten. Thanks!

Comment thread ml_peg/analysis/molecular/CRBH20/analyse_CRBH20.py Outdated
Comment thread ml_peg/analysis/molecular/CRBH20/analyse_CRBH20.py Outdated
Comment thread ml_peg/analysis/molecular/CRBH20/analyse_CRBH20.py Outdated
Comment thread ml_peg/app/molecular/CRBH20/app_CRBH20.py Outdated
Comment thread ml_peg/calcs/molecular/CRBH20/calc_CRBH20.py Outdated
Comment thread ml_peg/calcs/molecular/CRBH20/calc_CRBH20.py
Comment thread ml_peg/calcs/molecular/CRBH20/calc_CRBH20.py Outdated
Comment thread ml_peg/calcs/molecular/CRBH20/calc_CRBH20.py Outdated
Comment thread ml_peg/calcs/molecular/CRBH20/calc_CRBH20.py Outdated
Comment thread ml_peg/analysis/molecular/CRBH20/analyse_CRBH20.py Outdated
Comment thread ml_peg/analysis/molecular/CRBH20/analyse_CRBH20.py Outdated
Comment thread ml_peg/analysis/molecular/CRBH20/analyse_CRBH20.py Outdated
Comment thread ml_peg/analysis/molecular/CRBH20/analyse_CRBH20.py Outdated
Comment thread ml_peg/analysis/molecular/CRBH20/analyse_CRBH20.py Outdated
Comment thread ml_peg/analysis/molecular/CRBH20/analyse_CRBH20.py Outdated
Comment thread ml_peg/analysis/molecular/CRBH20/analyse_CRBH20.py Outdated
Comment thread ml_peg/analysis/molecular/CRBH20/analyse_CRBH20.py Outdated
Comment thread ml_peg/analysis/molecular/CRBH20/metrics.yml Outdated
Comment thread ml_peg/analysis/molecular/CRBH20/metrics.yml Outdated
Comment thread ml_peg/app/molecular/CRBH20/app_CRBH20.py Outdated
Comment thread ml_peg/app/molecular/CRBH20/app_CRBH20.py Outdated
Comment thread ml_peg/app/molecular/CRBH20/app_CRBH20.py Outdated
Comment thread ml_peg/app/molecular/CRBH20/app_CRBH20.py Outdated
Comment thread CRBH20.zip Outdated
Co-authored-by: Joseph Hart <92541539+joehart2001@users.noreply.github.com>
Co-authored-by: Elliott Kasoar <45317199+ElliottKasoar@users.noreply.github.com>

@Quantumplations Quantumplations left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accepted several of the simpler comments, will take a closer look at the suggestions.

Quantumplations and others added 2 commits July 8, 2026 23:25
- Catch calculation errors and store NaN energies, following the element
  filtering guidelines; verified with MockErrorCalculator
- Set default charge and spin before calculating energies
- Replace prints with tqdm progress bar and remove commented-out code
- Define single eV to kcal/mol conversion via ase.units
- Store references directly, propagate NaNs instead of filtering them
- Save elemental info for filtering via get_struct_info, and pass
  info_path to the app
- Add CRBH20 documentation and link it from the app
- Remove CRBH20.zip, now hosted on S3

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Comment thread ml_peg/analysis/molecular_reactions/CRBH20/metrics.yml Outdated
- Move calc, analysis, and app modules from molecular to
  molecular_reactions, alongside the RDB7 barrier benchmark
- Update data, output, and asset paths to the new category
- Add molecular_reactions documentation page and register it in the
  benchmarks toctree; point the app docs link at it
- Record level_of_theory (r2SCAN) in metrics.yml, matching sibling
  benchmarks
ElliottKasoar
ElliottKasoar previously approved these changes Jul 10, 2026

@ElliottKasoar ElliottKasoar left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apart from @joehart2001's comment on the hover text to be resolved, I think this is good to go, thanks for all the hard work!

@ElliottKasoar ElliottKasoar merged commit 7d2f985 into ddmms:main Jul 10, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new benchmark Proposals and suggestions for new benchmarks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmarking CRBH20 (Chemical Reaction Barrier Heights) dataset against DFT

3 participants