test_gradgrad fails on cpu and gpu

[ifeherva]

I built the library just fine, all unit tests complete except test_gradgrad().

The error I get is 'Numerical gradient for function expected to be zero'.

Note, I am building this on a host that has no GPU.

Using Pytorch 1.6

[ifeherva]

Just tested on gpu and the same error occurs. I am wondering if this impacts the usage of the operator.