Muon for multi-head attention weights

[schmidt-ai]

Should the weights for separate attention heads be separated before applying muon? I ask because I see in the blog post that "Muon works better for optimizing transformers if it is applied to their Q, K, V parameters separately", which makes me wonder there would be any benefit going further by splitting out the heads, too.

[parlance-zz]

Thanks for the info, I didn't know splitting QKV into separate matrices made a difference with muon.

This makes me wonder about conv params using groups > 1; each group could be separated as individual matrices. Time to do some science I guess.