weights become very large and then loss = nan

[chyh1990]

I use caffe to train my CNN but loss become nan after a few thousand iterations.
I dump the weights before that iterations and I found that some weights in the inner product layers become very large (e.g. +3e29). I check my shuffled input data and make sure they are in reasonable range.

I represent this in both GPU mode and CPU mode. Does this indicate some numeric problems in caffe or other causes?

[aravindhm]

I had similar issues when the learning rate was too high for my network. Lowering the learning rate solved the problem for me (I don't think it was a numerical problem in caffe). An indicator for high learning rate is a diverging error value and this typically leads to NaN in my experiments.

[chyh1990]

I don't think it is the learning rate's problem. I dump out the weight of my network and find that only one of the bias value in the last inner product layer become 3e+29 after a specific iteration, but all other weights seems good (e.g. between -10.0 and 10.0). Strange...

[smiley19]

@chyh1990 I think I might have same problem with you.
But I don't know how to check all the weights and bias in the model.
I am a beginner in python. I try the example for filter_visualization.ipynb but there are some errors.
Could you tell me how to get all parameters in my training model? Thanks a lot!

[sguada]

Try to set initial bias to 0.1 in all layers, or add regularization to the
bias (just set all the weight_decays in the layers to 1)

On Tuesday, May 13, 2014, smiley19 notifications@github.com wrote:

@chyh1990 https://github.com/chyh1990 I think I might have same problem
with you.
But I don't know how to check all the weights and bias in the model.
I am a beginner in python. I try the example for
filter_visualization.ipynb but there are some errors.
Could you tell me how to get all parameters in my training model? Thanks a
lot!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/373#issuecomment-42984262
.

Sergio

[smiley19]

@chyh1990 Thanks for your help, you do gave me a big favor!

[shelhamer]

Re: #373 (comment) about WriteProtoToTextFile, this is fixed in dev by #417.

[Yangqing]

(I assume this has been addressed, but feel free to reopen the issue should there be further questions.)

[henuwpf]

@chyh1990 Could you tell me how to get all parameters in my training model? Thanks very much!

[hyojinie]

I understand that regularization gets rid of unreasonable values for biases and weights, but I wonder how could a loss become NaN? Is it because those values resulted the loss to be -inf?