Question about memory consumption

[zhiqiangdon]

Hi @ldeecke,

Thanks for sharing the code.

I am trying to use your mode normalization in Wideresnet with depth 28 and widen factor 10. Training on CIFAR datasets with batch size 128 and batch norm, it takes 5983M memory. However, when I replace the batch norm with your mode norm, it is out of memory in a 16G V100 GPU. Could you give some analysis on why it consumes so much memory in training? Any suggestions to save memory?

Thanks

[ldeecke]

Hi @zhiqiangdon, thanks for trying out mode norm, and apologies for the late response, I somehow overlooked this!

The efficiency of this version here is indeed an issue. My goal is to address it eventually, but so far I've unfortunately never found the time required to do it.

One thing that has been very successful for us in the past is to use mode norm in early layers of the network (see also 2. of the readme.MD): we noticed that networks become domain-insensitive (or mode-insensitive) at deeper layers. Mode norm accelerates this, so networks become domain-invariant earlier, and can focus more of their capacity on the actual task. Depends on what you would be using mode norm for, of course, but this has worked very well for us!