Bug in LayerNorm?

[feixh]

Hi there,

I feel like this line is not correct:
https://github.com/szymanowiczs/viewset-diffusion/blame/c25c94d310268b98ba4c07f6307b593cc644e568/model/unet_parts.py#L139

which, in my opinion, should be
return (x - mean) / (var + eps).rsqrt() * self.g.

Or, did I miss something?

[szymanowiczs]

Yes, you're right - thanks for pointing out.

It's the same couple of lines down in LayerNorm2D. I inherited this bit from https://github.com/lucidrains/denoising-diffusion-pytorch/blob/05bbae8ce12ab85d1729ff7af11b0d20b0abad3f/denoising_diffusion_pytorch/denoising_diffusion_pytorch.py#L120C55-L120C55

and didn't notice.

That being said, the models train fine and my existing checkpoints use the formulation I have now, so I don't want to make the change just yet. I'll investigate if it affects perf if I get some time.