Dear HamGNN developer,
I’m training HamGNN and noticed a large numerical precision-related issue.
After 1st training is finished with hamilitonian MAE of 1e-5, I tried to train/evaluate with FP32 for 2nd training, the MAE for both band energy and bandgap was as high as a few tens of meV. I tried several settings, but never obtained better MAE in band energy/gap.
However, when I switch the training to FP64, both MAE values improve by roughly 100 times reaching to sub meV level.
This makes me suspect that some part of the Hamiltonian / overlap / eigenvalue calculation may be numerically unstable in FP32.
However, I noticed that the default setting is FP32 and your paper looks like to use FP32, which makes me confused.
There are two guesses:
One thing is that I have used PBE0 for the training dataset, and the CBM was highly sensitive to the PBE0 setting.
I doubt that the HamGNN cannot well capture the hybrid functional hamiltonian with FP32.
Another one is that I modified the code by adding a small jitter to Overlap_k:
S_k = S_k + jitter * I
I added it to resolve the negative value problem of S(r) file from ABACUS calculation which maybe the issue you announced in the tutorial.
The jitter value is very small (1e-10), so I’m not sure if it can meaningfully affect FP32 results. But I'm not sure whether this is the announced prediction error or not. My current guess is that the issue may not be the jitter itself, but rather that 'Overlap_k' or the generalized eigenproblem is ill-conditioned in FP32.
I also noticed that GPU memory usage is quite high even when using FP32. With FP64, the memory usage increases further and becomes impossible to train even with high-end GPU like H200.
Could you help clarify the issues?
Any guidance on the recommended precision settings or memory-saving configuration would be greatly appreciated.
Thanks!
Dear HamGNN developer,
I’m training HamGNN and noticed a large numerical precision-related issue.
After 1st training is finished with hamilitonian MAE of 1e-5, I tried to train/evaluate with FP32 for 2nd training, the MAE for both band energy and bandgap was as high as a few tens of meV. I tried several settings, but never obtained better MAE in band energy/gap.
However, when I switch the training to FP64, both MAE values improve by roughly 100 times reaching to sub meV level.
This makes me suspect that some part of the Hamiltonian / overlap / eigenvalue calculation may be numerically unstable in FP32.
However, I noticed that the default setting is FP32 and your paper looks like to use FP32, which makes me confused.
There are two guesses:
One thing is that I have used PBE0 for the training dataset, and the CBM was highly sensitive to the PBE0 setting.
I doubt that the HamGNN cannot well capture the hybrid functional hamiltonian with FP32.
Another one is that I modified the code by adding a small jitter to Overlap_k:
S_k = S_k + jitter * I
I added it to resolve the negative value problem of S(r) file from ABACUS calculation which maybe the issue you announced in the tutorial.
The jitter value is very small (1e-10), so I’m not sure if it can meaningfully affect FP32 results. But I'm not sure whether this is the announced prediction error or not. My current guess is that the issue may not be the jitter itself, but rather that 'Overlap_k' or the generalized eigenproblem is ill-conditioned in FP32.
I also noticed that GPU memory usage is quite high even when using FP32. With FP64, the memory usage increases further and becomes impossible to train even with high-end GPU like H200.
Could you help clarify the issues?
Any guidance on the recommended precision settings or memory-saving configuration would be greatly appreciated.
Thanks!