Save layer connections as a parameter so that it can be reloaded

[johanjino]

Hi,

I was using the difflogic experiments as shown in repository. I noticed only the probability distribution of logic gates (i.e weights) of each neuron is saved. Which means reloading weights will not give same accuracy again, since the connections may be initialized differently.

difflogic/difflogic/difflogic.py

Line 54 in 469702c

self.indices = self.get_connections(self.connections, device)

To solve this I propose saving indices as self.indices = torch.nn.parameter.Parameter(torch.stack(self.get_connections(self.connections, device)), requires_grad=False). This may need relevant changes in other uses of self.indices i.e. in the forward pass.

Note: In the give experiments the torch/numpy/random seed is set initially which means connections are initialized in a same manner every time. Which is likely why #11 was able to resume training without much issue.

For me, I was unable to reproduce the accuracy of a GPU trained model weights on CPU... and there I noticed the indices are being initialized differently.

If this seems reasonable, I am happy to create a PR for it 😄.

[SuperrMurlocc]

@johanjino I also had to do something with indices in my research to be able to save and load them.
Since indices is not something to be trained, rather than using Parameter, I'd suggest using register_buffer.
As I found here, this is what buffer is used for 😄