Weight dropout

[Judyxujj]

This PR adds the weight dropout

[albertz]

I think following torch.nn.utils.weight_norm or torch.nn.utils.parametrizations.weight_norm for this makes more sense, or not? This doesn't need to be (or rather should not be) specific for one module (here it is specific to torch.nn.Linear) but it should be generic, and less hacky, and simpler.

[Judyxujj]

I think following torch.nn.utils.weight_norm or torch.nn.utils.parametrizations.weight_norm for this makes more sense, or not? This doesn't need to be (or rather should not be) specific for one module (here it is specific to torch.nn.Linear) but it should be generic, and less hacky, and simpler.

@albertz but here we can also use the WeightDrop class to wrap other torch.nn.Module like this

gru = torch.nn.GRUCell(2, 2)
weights = ['weight_hh']
weight_drop_gru = WeightDrop(gru, weights, dropout=0.9)

I added WeightDropLinear because I think it would be most frequently used.

[albertz]

Ah sorry I misread the code regarding nn.Linear. You are right, your WeightDrop is generic. But I still think it's too complicated and too hacky and non-PyTorch like compared to how torch.nn.utils.weight_norm or torch.nn.utils.parametrizations.weight_norm works. I would follow standard PyTorch code (like torch.nn.utils.weight_norm or torch.nn.utils.parametrizations.weight_norm), i.e. add weight dropout just in the same way.