Difficulty Training Multi-Layer MHN with Custom Tau Settings

[WOOYULJUNG]

I attempted to train a single hidden layer MHN using this library. However, the library does not allow setting tau for each layer—it is fixed at 1. According to the HAM paper, this is not appropriate. In the single hidden layer case, training proceeds quite well. However, when extending to a multi-layer architecture, the tau values need to be set differently for each layer. When I extend to a multi-layer model and arbitrarily set tau values to train using backpropagation, the training does not work properly.

If anyone has successfully trained a multi-layer model using this library, or if anyone has experimented with varying tau in a single hidden layer and observed successful training, I would greatly appreciate any insights or tips on how to set tau and the number of time steps used in training (nsteps).

[cgoemaere]

Hey @WOOYULJUNG! I'm not affiliated to this repo, but I stumbled upon your issue and thought I could help! I have trained several multi-layer HAMs for my paper, but I also ran into some troubles, so I can sympathize ;)

Setting tau=1 is actually the best choice. It gives you the fastest convergence, while maintaining stability (generally speaking). This is possible because in HAMs, the equilibrium value of a state typically depends only on its neighbors — not itself!

State convergence is always attainable, but you may need a lot of iterations. You can speed things up with even-odd splitting or DEQ solvers (see the linked paper: it's literally about this stuff). The number of steps required depends on the problem, but you can track the so-called 'relative residual' to see how well you're doing (again, see paper).

Weight convergence is something else. When training deeper models, I was struggling with vanishing gradients and unstable training. In the end, I used the Madam optimizer with a lr warmup schedule. It's not pretty, but it works. There's definitely a better way of doing it, but I never figured it out.

Finally, I should probably mention that I didn't work with MHNs specifically (i.e., HAMs with softmax activations). From work of a colleague, I do know that these act weird when used iteratively, so I can imagine stacking them makes things even worse. IIRC, the hamux library sees them more as a state regularization term than as actual layers (since they're only operating on a single state). Overall, my intuition is that stacking MHNs sounds like a bad idea.