A question about the pred noise.

[northagain]

Thanks for ur excellent work!
I have a question about ddim loss which is computed with noise and pred noise.
The pred noise, which is the output of ScheduledCNNRefine Model is >0, because the last activation function is RELU.
However, shouldn't the randn noise contains both >0 parts and <0 parts?

[duanyiqun]

It is a good question. It should be theoretically right in this case.
Have you tried to use LeakyRelu instead for training? Limited to my memory, I didn't remember a clear difference. But I would appreciate it if you have some comparison results between ReLU and LeadkyReLU. If the results are better, I may have an update. Or we may have a pull request together.