training process halt in few epochs

[tzzcl]

When I use the given example to train on ResNet, It will sometimes crash on some iterations, not pushing to the next sub batch.

I use these command to train the network:
CUDA_VISIBLE_DEVICES=12,13,14,15 OMP_NUM_THREADS=4 th main.lua -data /opt/xxx/images/ -resume checkpoints -batchSize 128 -nThreads 8 -nGPU 4 -shareGradInput true

is there anybody encounter the same problem?

[colesbury]

Do you have a stack trace from when it crashes?

[DmitryUlyanov]

I have a similar issue, in my case there is completely no errors popping out. It just freezes.