Why is there a small coefficient for the standard deviation in the reparameterization trick?

[Isuxiz]

CVAR/model/warm.py

Lines 362 to 363 in e882d50

	z = mean + 1e-4 * torch.exp(log_v * 0.5) * torch.randn(mean.size()).to(self.device)
	z_p = mean_p + 1e-4 * torch.exp(log_v_p * 0.5) * torch.randn(mean_p.size()).to(self.device)

1. What is the meaning of 1e-4 here?
2. (Critical question) Does this restriction on the standard deviation weaken the distribution so that it degenerates to a fixed-point representation of the mean?

I removed the coefficient of 1e-4 (i.e. the coefficient was changed to 1) and conducted the experiment again, found that the effect dropped significantly.

[xianghang]

@Isuxiz hello, are you still looking into this? I find another interesting thing to try would be change 1e-4 to 0 .

[Isuxiz]

@Isuxiz hello, are you still looking into this? I find another interesting thing to try would be change 1e-4 to 0 .

Yes, I've also discovered this problem.
The explanation I gave myself is that this is something similar to the concept of "temperature" in other formulas (such as infoNCE), which is used to control the strength of randomness.
Changing this value to 0 actually makes VAE lose its randomness to a certain extent, since it will only use mean as z.