DIffusion Sampler slow learning - possible bug

[josephdviviano]

For example, at epoch 1999,
our implementation gives a sampler like this:

and the base implementation gives:

Our implementation shows a similar mode-collapse behavior after >= 4000 epochs.

Originally posted by @hyeok9855 in #419 (comment)