EMAcodebook update

[GuGuGuDove]

Hi maxjcohen,

in codebook.py line 186, if the ema_positions got divided by ema_cluster_size, that means the first term in line 183 got divided too, which seems not reasonable? (only the 2nd term in line 183 is the ensemble of several similar encodings so normalization needed but the 1st term is not)

Could you give me some hints? Thx!

[maxjcohen]

Hi, and thank you for taking an interest in this repo.

When you say

the ema_positions got divided by ema_cluster_size

I don't quite follow what you mean, as it is not the ema_positions values which are updated themselves, but the codebooks:

self.weight.data = self.ema_positions / self.ema_cluster_size.unsqueeze(-1)

As described in the last page of the original paper (only the arXiv version).