NaN GradNorm. Exiting

[chrshdl]

In my attempt to train a model using a GTX1070 8GB I am getting an error "NaN GradNorm. Exiting". I was using a batch size of 128. The training set consists of olivia.zip and the librispeech corpus of clean-360.

Previsouly I was experimenting with batch sizes between 256 and 512 and I was getting CUDA_OUT_OF_MEMORY errors. I couldn't eliminate those errors, configuring the Tensorflow session to only use 90% of the available GDRAM didn't bring any remedy.

        sess_conf.gpu_options.allow_growth = True
        sess_conf.gpu_options.per_process_gpu_memory_fraction = 0.9

Do you have any hints how to overcome this error?

[chrislengerich]

We used minibatch 256 on Titan Xs (12GB memory), but there's no reason you couldn't use something as low as 32, which will address the OOM errors. The gradient norm exploding could be a number of reasons, but you could try a lower learning rate, random initialization and check your data format and preprocessing for the librispeech corpus to start.

[chrshdl]

Hi Chris and thanks for the quick response! It seems that I'm not equipped with adequate hardware. I tried your suggestion and even using a batchsize of 8 leads to out of memory errors. Would it be possible to share your trained model? Thanks in advance.

[chrislengerich]

You also change some of the hyperparameter settings to change memory usage as well: https://github.com/mindorii/kws/blob/master/configs/kws.json. Probably start with hidden_size or num_layers. A pre-trained model is a good idea - will think about it.

[chrshdl]

Thanks! In the meantime I was able to train a model on my GTX1070 8GB using the following params:

    "data" : {
        "train_jsons" : [
            "data/olivia_train.json",
            "data/olivia_train_noise.json",
            "data/train_clean_360.json"

        ],
        "sample_rate" : 8000
    },

    "inference" : {
        "hidden_size" : 64,
        "num_layers" : 3,
        "num_filters" : 32,
        "temporal_context" : 15,
        "frequency_context" : 32,
        "padding" : "VALID",
        "stride" : 3,
        "batch_size" : 32
    },

    "train" : {
        "learning_rate" : 1e-4,
        "lr_decay_steps" : 5000,
        "lr_decay_rate" : 0.9,
        "momentum" : 0.95,
        "max_grad_norm" : 100,
        "epochs" : 120
    }

Additionally, to limit the audio to a keyword-like duration, I altered the max_duration to 4.0 in the constructor of AudioProducer. Also the olivia-dataset was resampled to a 16000 rate.

The AvgCost I get after 120 epochs is around 20. As shown in the snippet below, there is a rise of the cost between iterations, from 0.1 to 60, sometimes even to 120. Do you have a hint how to overcome this? Thanks again for your effort. Really appreciate it!

Iter 80860: AvgCost 19.4390125275, Cost 0.09, GradNorm 1.39, CumTime 51.70 (s), CompTime 50.00 (s), AvgItTime 0.08 (s)
Iter 80861: AvgCost 19.2455310822, Cost 0.10, GradNorm 7.39, CumTime 51.76 (s), CompTime 50.05 (s), AvgItTime 0.08 (s)
Iter 80862: AvgCost 19.0540733337, Cost 121.74, GradNorm 120.73, CumTime 51.91 (s), CompTime 50.20 (s), AvgItTime 0.08 (s)
Iter 80863: AvgCost 20.0808925629, Cost 0.09, GradNorm 1.76, CumTime 51.96 (s), CompTime 50.25 (s), AvgItTime 0.08 (s)
Iter 80864: AvgCost 19.881029129, Cost 119.98, GradNorm 67.08, CumTime 52.10 (s), CompTime 50.38 (s), AvgItTime 0.08 (s)
Iter 80865: AvgCost 20.8819999695, Cost 65.30, GradNorm 46.41, CumTime 52.20 (s), CompTime 50.48 (s), AvgItTime 0.08 (s)
Iter 80866: AvgCost 21.3261814117, Cost 0.08, GradNorm 0.56, CumTime 52.27 (s), CompTime 50.55 (s), AvgItTime 0.08 (s)
Iter 80867: AvgCost 21.1137504578, Cost 0.13, GradNorm 6.55, CumTime 52.37 (s), CompTime 50.65 (s), AvgItTime 0.08 (s)
Iter 80868: AvgCost 20.9039535522, Cost 0.11, GradNorm 2.47, CumTime 52.45 (s), CompTime 50.72 (s), AvgItTime 0.08 (s)
Iter 80869: AvgCost 20.6960010529, Cost 4.99, GradNorm 37.74, CumTime 52.50 (s), CompTime 50.77 (s), AvgItTime 0.08 (s)
Iter 80870: AvgCost 20.5389900208, Cost 109.71, GradNorm 81.05, CumTime 52.63 (s), CompTime 50.89 (s), AvgItTime 0.08 (s)
Iter 80871: AvgCost 21.4307289124, Cost 0.10, GradNorm 2.75, CumTime 52.67 (s), CompTime 50.94 (s), AvgItTime 0.08 (s)
Iter 80872: AvgCost 21.2174167633, Cost 0.11, GradNorm 4.82, CumTime 52.73 (s), CompTime 50.99 (s), AvgItTime 0.08 (s)
Iter 80873: AvgCost 21.0063419342, Cost 0.20, GradNorm 63.43, CumTime 52.78 (s), CompTime 51.05 (s), AvgItTime 0.08 (s)
Iter 80874: AvgCost 20.7982883453, Cost 88.11, GradNorm 54.49, CumTime 52.88 (s), CompTime 51.15 (s), AvgItTime 0.08 (s)
Iter 80875: AvgCost 21.4713802338, Cost 116.29, GradNorm 110.63, CumTime 53.03 (s), CompTime 51.28 (s), AvgItTime 0.08 (s)
Iter 80876: AvgCost 22.4195976257, Cost 0.09, GradNorm 2.19, CumTime 53.08 (s), CompTime 51.34 (s), AvgItTime 0.08 (s)
Iter 80877: AvgCost 22.1963043213, Cost 0.08, GradNorm 0.57, CumTime 53.14 (s), CompTime 51.40 (s), AvgItTime 0.08 (s)
Iter 80878: AvgCost 21.9751548767, Cost 0.10, GradNorm 2.36, CumTime 53.21 (s), CompTime 51.46 (s), AvgItTime 0.08 (s)
Iter 80879: AvgCost 21.7563648224, Cost 0.09, GradNorm 1.82, CumTime 53.28 (s), CompTime 51.53 (s), AvgItTime 0.08 (s)
Iter 80880: AvgCost 21.5396842957, Cost 0.08, GradNorm 0.61, CumTime 53.35 (s), CompTime 51.60 (s), AvgItTime 0.08 (s)
Total time:  53.4438791275
Finished epoch 119

[xxxxlr]

@chrishedel ,
Seems like I am having the same issue as you do for both explosion and grad jumping around :)...

However, I couldn't resolve either.
I tried your config, but no luck and wondering if it's the envirment setup causing it.

I am using ubuntu 14.04, tf 1.4, CUDA 8. (although the suggested env is CUDA 7.5, but that seems like not a compatible option from tf official website).
What envirment were you using?

Thanks!

[chrshdl]

Hi @xxxxlr ,

I trained on an ubuntu 16.04 machine, not sure about the TF and CUDA versions though. I would focus on training on a small dataset fist, you can start by only using https://github.com/mindorii/kws#data with a batchsize of 16 or 8.

Hope this helps.

[xxxxlr]

Thanks for the info @chrishedel

[lypenghao]

@ chrshdl
Can you share your train_clean_360.json file with me,.Thank you
wangpenghao@aliyun.com