Lack of Diversity in Generated Images in Two-Stage Diffusion Model

[songjijun]

Description:
I am currently working with a two-stage diffusion model consisting of two separately trained stages: guided diffusion (first stage) and SinDiffusion (second stage). During inference, the process is divided into 1000 timesteps, where 600 timesteps are used for the guided diffusion stage and the remaining 400 timesteps for SinDiffusion.

However, I am facing the following issues:

• When using both stages together during inference, the generated images show very limited diversity, often producing only one or two types of images (compared to the original dataset).
• If I use only the guided diffusion stage for inference, the generated images exhibit significantly more diversity.
• When using only the SinDiffusion stage for inference, the generated images are of a single type, completely lacking diversity.

Steps to Reproduce:

1. Train the two stages separately (guided diffusion and SinDiffusion).
2. Perform inference using both stages together (600 timesteps for guided diffusion and 400 timesteps for SinDiffusion).
3. Observe that the generated images lack diversity, typically producing only one or two types of images.
4. Perform inference using only guided diffusion or only SinDiffusion and compare the results.

Expected Behavior:

• The generated images should retain the diversity of the original dataset, especially when using both stages during inference.

Actual Behavior:

• When using both stages together, the generated images show only one or two types of images.
• When using SinDiffusion alone, it generates only a single type of image, lacking diversity.

[AndysonYs]

Hi, thanks for sharing the issue.

First, I’d like to clarify—are you using our codebase for these experiments, or are you using a different implementation of guided diffusion and SinDiffusion?

If you’re using our code on MVTec, then the behavior you described when running only the SinDiffusion stage doesn’t match our experience. Specifically, SinDiffusion alone should generate diverse outputs, often with significant variation and sometimes even struggling to retain structural consistency. Please double-check your inference setup to ensure everything is configured correctly.

Let me know more details and I’ll be happy to help further.