NaN values in Predictions

[nhafez]

After following the instructions in the latest commit and then running the train_and_test_deepim_all.sh I got the following error:

Traceback (most recent call last):
File "experiments/deepim/deepim_train_test.py", line 20, in
train.main()
File "experiments/deepim/../../deepim/train.py", line 287, in main
config.TRAIN.begin_epoch, config.TRAIN.end_epoch, config.TRAIN.lr, config.TRAIN.lr_step)
File "experiments/deepim/../../deepim/train.py", line 280, in train_net
prefix=prefix)
File "experiments/deepim/../../deepim/core/module.py", line 1026, in fit
data_batch = interBatchUpdater.forward(data_batch, preds, config)
File "experiments/deepim/../../lib/pair_matching/batch_updater_py_multi.py", line 231, in forward
rot_type='QUAT')
File "experiments/deepim/../../lib/pair_matching/RT_transform.py", line 34, in calc_RT_delta
r = mat2quat(Rm_delta)
File "experiments/deepim/../../lib/pair_matching/RT_transform.py", line 459, in mat2quat
vals, vecs = np.linalg.eigh(K)
File "/home/saadhana/.local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 1410, in eigh
w, vt = gufunc(a, signature=signature, extobj=extobj)
File "/home/saadhana/.local/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 95, in _raise_linalgerror_eigenvalues_nonconvergence
raise LinAlgError("Eigenvalues did not converge")
numpy.linalg.linalg.LinAlgError: Eigenvalues did not converge

Looks like the predicted poses are all NaN values.
I printed the rotation and translation predicted:

[array([[nan, nan, nan, nan],
[nan, nan, nan, nan],
[nan, nan, nan, nan],
[nan, nan, nan, nan]], dtype=float32)] [array([[nan, nan, nan],
[nan, nan, nan],
[nan, nan, nan],
[nan, nan, nan]], dtype=float32)]

Has anybody successfully trained the network for LINEMOD or OCCLUSION datasets?

[liyi14]

I executed train_and_test_deepim_all.sh and doesn't found such error. Can you provide more information, like the context of this error? Does it fail in the first iteration or after a few iterations?

[nhafez]

For the first batch it completes one iteration and on the next one this error happens because the predictions are NaN

[liyi14]

Can you change the frequent in the experiments/deepim/cfg/*_any/all.yaml(abbreviate as config below)->default to 1 and tell me the result running such modifications separately:

1. rerun using the train_test_deepim_all.sh
2. run the train_test_deepim_ape.yaml
3. change train_iter_size in config->network to 1 and run any config reporting error before
4. replace dataset: LM6D_REFINE+LM6D_REFINE_SYN to dataset: LM6D_REFINE and image_set: train_+train_ to image_set: train_
5. change the config->TRAIN->warmup_lr to 0.0
   Tell me what happened after applying such modifications, thank you.