Home

Weight Normalization complete implementation code:

The implementation contains all the weight normalization codes from the article.

Some thoughtful things:

• The code uses templates. The weight-norm part and basic batch-norm part was straightforward to implement. For the mean only batch-normalization part the pop_mean variable was moved out as a global variable both in conv2d(...) and dense(...) functions. The reason for that was as for weight normalization, the initialization was defined. But the mean only batch-normalization function, and for this the pop_mean variable had to be constructed after initialization. In initialization step the pop_mean variable was not set and not created (the graph wont be know about the existence). So I moved this out as a "global" variable in both the conv2d(...) and dense(...) functions, and passed as a parameter for mean_only_batchnormalization_impl function(...). This way the pop_mean variable was created at before initialization step, and this could be reused when the training phase comes. But for this I had to know the dimension for pop_mean variable: which' shape is [num_filters] for conv2d and [num_units] 1D vectors for the dense layers.

• Also in the mean_only_batchnormalization_impl function(...): the pop_mean_op which is responsible for updating the 'batch_mean' was needed to pass within square brackets for tf.control_dependencies(...) function, unless it gives a "tensor iteration fault" like failure. So this is solved now.