About FLUX Baseline Performance

[haoweiz23]

Hi, thanks for your great work.

Based on your code in Cache4Diffusion/flux/speca-flux, I ran the baseline of FLUX (1024x1024, guidance=3.5) and obtained the following results, which differ from those reported in your SpeCa paper. Do you have any suggestions or insights on this discrepancy?

Also, I noticed that the ImageReward score at 25 steps is even higher than at 50 steps. Have you observed similar behavior in your experiments?

Method	FLOPs (TFLOPs)	ImageReward	CLIP-Score
Original 50 steps (baseline)	2975.99	0.946253	30.6349
Original 25 steps	1488.00	0.955330	30.9385
Original 20 steps	1190.40	0.925985	30.8404
Original 15 steps	892.80	0.920931	30.6974
Original 10 steps	595.20	0.885935	30.2843

[Tammytcl]

Hi, thanks for your detailed feedback and for reproducing the FLUX baseline.

Regarding the discrepancy in ImageReward scores — indeed, the metric can show slight variations across different devices, but in our experiments the gap was not as large as what you observed. One important detail to note is that in our evaluation, the ImageReward measurement uses seed=0 instead of None. During inference, the random seed is incremented with the image index, ensuring consistent sampling behavior. You might want to double-check that setup on your side.

As for the FLOPs difference, that’s expected. If you used the standard Python cal_flops library, it will skip the computation from the efficient attention blocks used in FLUX. In our experiments, we implemented a modified FLOPs calculation tool that accounts for these efficient attention operations, so the numbers won’t align directly.

If the ImageReward discrepancy remains after checking the seed configuration, feel free to share your inference script — we’ll be happy to take a closer look.

[haoweiz23]

@Tammytcl Thank you for sharing. As you mentioned, does cal_flops account for only a portion of the FLOPs? Would you mind sharing your modified FLOPs calculation tool?

[Tammytcl]

For now, we don’t plan to open-source our internal evaluation package. We will release all our metric evaluation components for different models together in our upcoming Diffusion Cache survey project.

We have already released our initial survey on Diffusion Cache research:
https://arxiv.org/pdf/2510.19755

Later this year, we plan to build a public leaderboard and open-source all our evaluation methods and scripts across different diffusion models.

In the meantime, you can refer to our TaylorSeer repository:
https://github.com/Shenyi-Z/TaylorSeer

In the FLUX section, there is a --test_flops option that enables FLOPs evaluation mode.
However, please note that you also need to manually disable the efficient attention module (e.g., by switching off the attention implementation) — otherwise, the FLOPs of the attention layers will still be skipped during measurement.
Our uploaded code already supports this configuration, allowing you to accurately evaluate FLOPs.