Torch epoch rounding issue

[lukeyeager]

Here is the output from torch training AlexNet:

2015-05-21 14:13:15 [INFO ] Training (epoch 0): loss = nan, lr = 0.01
2015-05-21 14:15:55 [INFO ] Training (epoch 0.01): loss = 0.99398752933095, lr = 0.01
2015-05-21 14:18:31 [INFO ] Training (epoch 0.01): loss = 1.0189151917673, lr = 0.01
2015-05-21 14:21:05 [INFO ] Training (epoch 0.02): loss = 0.9794481931978, lr = 0.01
2015-05-21 14:23:39 [INFO ] Training (epoch 0.02): loss = 1.0302608635775, lr = 0.01
2015-05-21 14:26:11 [INFO ] Training (epoch 0.02): loss = 0.98033467011087, lr = 0.01

DIGITS doesn't know what to do with two training loss values for epoch=0.01, so nothing gets displayed in the graph. We need to add more significant digits to the epoch so each update has a unique value.

[Sravan2j]

@lukeyeager: fixed the issue. Please test it once and close issue if fix looks good. Thanks for your time.

[lukeyeager]

I think the main issue is probably fixed. But when testing it, I tried to run LeNet on MNIST with training_epochs=1 and val_interval=0.0000001. Now no validation is done at all for the model. When I do the same with caffe, I get test_inverval=1 in the solver, which translates to validation being done after each iteration.

You may already be doing this for torch, but when I check the task arguments, I see --interval=0.000000, so there's a rounding issue somewhere in there as well.

NOTE: I realize this is an edge case, but it's one that I actually use pretty often when testing things. I like to be able to enter 0.00000000001 for val_interval and know that I'm going to get validation after each iteration.

[gheinrich]

I noticed that in the Caffe implementation the training loss is reported on first iteration. In the Torch implementation the training loss is only reported after the first interval=min(5000,dataset_size/8). Since the data structure in train.py is only created after a training loss update, it takes possibly much longer to see the first update in the web UI when training a Torch network. This is especially true with a very small validation interval as this slows down training a lot.
A workaroud for this issue is to make Torch behave like Caffe i.e. report the training loss after the first iteration.

[lukeyeager]

Yes, we should fix that somehow.

report the training loss after the first iteration

Sounds good to me.