Hi,
I've read the discussion per #15 and #50, but I still don't understand why the current sharedRmsprop impl avoids thread racing? Actually, the code still occasionally outputs NaN on my machine unless I set the thread be one. By tracking the error I can tell it is due to these two lines:
state.g:mul(momentum):addcmul(1 - momentum, dfdx, dfdx)
state.tmp:copy(state.g):add(epsilon):sqrt()
as state.tmp can become zero while being divided in the rest code and I guess the zeros are due to state.tmp:copy(state.g) from other thread where state.g happens to include 0s...
Meanwhile, by changing them to
state.g:mul(momentum):addcmul(1 - momentum, dfdx, dfdx)
state.tmp:sqrt(torch.add(state.g, epsilon))
the error seems to disappear.
It my modification reasonable? Or I just have to update OpenBLAS or something?
Hi,
I've read the discussion per #15 and #50, but I still don't understand why the current sharedRmsprop impl avoids thread racing? Actually, the code still occasionally outputs NaN on my machine unless I set the thread be one. By tracking the error I can tell it is due to these two lines:
as
state.tmpcan become zero while being divided in the rest code and I guess the zeros are due tostate.tmp:copy(state.g)from other thread wherestate.ghappens to include 0s...Meanwhile, by changing them to
the error seems to disappear.
It my modification reasonable? Or I just have to update OpenBLAS or something?