BatchInstanceNorm1d

[adrienchaton]

Hello,

I have been working on a project using a WAE with BatchNorm in the encoder and InstanceNorm in the decoder to have both representation/classification power for the encoder and the capacity of AdaIN for style conditioning in the decoder.

I choose this empirically throughout my experiments and then found out your paper which generalizes the possibility of mixing both.

I would like to try it and then make reference to your work. It was not a major feature of my research to separate the normalizations, just an implementation detail that I could replace with yours.

I replaced every 2d normalization in my networks by BatchInstanceNorm2d, no problem, it processes without error the 4D tensors (batch,chan,H,W).

However, I also had to change every 1d normalization by BatchInstanceNorm1d, which is for the 2D tensors output of linear layers. And there I have an error ...

I double checked the data, given a 2D tensor (90,1024) output of linear layer which is followed with BatchInstanceNorm1d(1024), I get the following:

/fast-2/adrien/virtualenv_p3/lib/python3.5/site-packages/torch/nn/functional.py in batch_norm(input, running_mean, running_var, weight, bias, training, momentum, eps)
1617 size_prods *= size[i + 2]
1618 if size_prods == 1:
-> 1619 raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
1620
1621 return torch.batch_norm(

ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 92160])

could you help fixing this please ?
ran on python 3.5 and pytorch 1.0.0

[adrienchaton]

I also tried switchable normalization from
https://github.com/switchablenorms/Switchable-Normalization

replacing normalization layers from my models with SwitchNorm1d and SwitchNorm2d
from https://github.com/switchablenorms/Switchable-Normalization/blob/master/devkit/ops/switchable_norm.py

I did not have issues with this implementation and I am currently training new models with SwitchNorm

If we can fix BatchInstanceNorm1d, then I can also train them and make comparison of both BatchInstanceNorm and SwitchNorm against the initial model with BN encoder and IN decoder

Thanks !

[adrienchaton]

I made a few modifications in the forward after # Instance norm
if case to not reshape 2D tensors that did not forward in BatchInstanceNorm1d
now it forwards so I will give it a try ... are these modifications correct to you ?

class _BatchInstanceNorm(_BatchNorm):
    def __init__(self, num_features, eps=1e-5, momentum=0.1, affine=True):
        super(_BatchInstanceNorm, self).__init__(num_features, eps, momentum, affine)
        self.gate = Parameter(torch.Tensor(num_features))
        self.gate.data.fill_(1)
        setattr(self.gate, 'bin_gate', True)

    def forward(self, input):
        self._check_input_dim(input)

        # Batch norm
        if self.affine:
            bn_w = self.weight * self.gate
        else:
            bn_w = self.gate
        out_bn = F.batch_norm(
            input, self.running_mean, self.running_var, bn_w, self.bias,
            self.training, self.momentum, self.eps)
        
        # Instance norm
        b, c  = input.size(0), input.size(1)
        if self.affine:
            in_w = self.weight * (1 - self.gate)
        else:
            in_w = 1 - self.gate
        
        # input = input.view(1, b * c, *input.size()[2:])
        # this did not forward in the following F.batch_norm(input,...) for 2D input tensors and 1d norm ..
        if input.dim() != 2:
            input = input.view(1, b * c, *input.size()[2:])
        
        out_in = F.batch_norm(
            input, None, None, None, None,
            True, self.momentum, self.eps)
        
        # out_in = out_in.view(b, c, *input.size()[2:])
        # same as before
        if input.dim() != 2:
            out_in = out_in.view(b, c, *input.size()[2:])
            out_in.mul_(in_w[None, :, None, None])
        else:
            out_in.mul_(in_w)

        return out_bn + out_in

[hyseob]

Hi, @adrienchaton
We didn't consider 1d case because Instance Normalization loses its effect in that case.
I.e., for a 2d tensor of (batch size, channels), Instance Normalization makes all elements become zero because the instance-wise mean equals the input tensor itself.
Nevertheless, it is worth making BIN capable of 1d case by disabling instance normalization (which is what Switchable Normalization did), which I will update soon.
Thank you!

[adrienchaton]

That is correct, I didn't take this into consideration.
Thank you for your answer, this update makes sense as for SN.