The results of MSCOCO FID are quite different from those of the paper

[huanpengchu]

As title, I based on https://github.com/mseitzer/pytorch-fid measuring fid, but PixArt alpha fid result was 25.264, do not tally with the paper (28.09). In addition, the fid of toca is 26.704, which is also inconsistent with the paper.
What's the reason, thank you

[huanpengchu]

clip score is normal

[Shenyi-Z]

Hello! Thank you for your interest in our work! We previously encountered a similar issue regarding manually reproducing the experimental results on PixArt-alpha, and we have provided a solution that you can refer to here: #8. The main reason for this discrepancy is the selection of different 30k images when constructing the NPZ file from the MSCOCO dataset. You may try using the NPZ file we provided for evaluation.

[huanpengchu]

Hello! Thank you for your interest in our work! We previously encountered a similar issue regarding manually reproducing the experimental results on PixArt-alpha, and we have provided a solution that you can refer to here: #8. The main reason for this discrepancy is the selection of different 30k images when constructing the NPZ file from the MSCOCO dataset. You may try using the NPZ file we provided for evaluation.

yes, I have used the npz which you shared in [#8]. The above results are based on this. I am very confusing. The FID is better than in the paper (the smaller the fid value, the better), and the values are inconsistent in the original model

[Shenyi-Z]

quite strange. We were able to reproduce the results from the paper on other devices, and the previous questioner in #8 also seems to have successfully reproduced them. Could you provide more detailed information so that I can reproduce the issue you encountered?

[huanpengchu]

Here's the command. Is there a problem？

torchrun --nproc_per_node=6 scripts/inference_ddp.py \
    --model_path /home/cache_exper/pixart_alpha/PixArt-XL-2-256x256.pth \
    --image_size 256 --bs 5 --txt_file /home/cache_exper/ToCa/COCO_caption_prompts_30k.txt \
    --fresh_threshold 3 --fresh_ratio 0.40 --cache_type attention --force_fresh global \
    --soft_fresh_weight 0.60 --ratio_scheduler ToCa

And the version of pytorch is as follows:
`torch 2.1.2

torchaudio 2.1.2

torchmetrics 1.6.1

torchvision 0.16.2

tornado 6.4.1

diffusers 0.32.2`

[Shenyi-Z]

Could it be due to a difference in soft_fresh_weight? In our recommended reproduction command, it is set to 0.25, but the difference shouldn’t be this significant... Since your results show a considerable deviation from the paper even for the original, non-accelerated model, you might consider adjusting the batch size to match our preset value of 100. Based on experience, batch size shouldn’t cause such a large difference either, but it can indeed affect the image content, possibly due to randomness.

[mgwillia]

I am also struggling to match what's in the paper; with the provided npz and prompts text text file, with PixArt-alpha 256x256, clean_fid with inception_v3 I get 24.85 FID. I also used batch size 100.

[Shenyi-Z]

I am also struggling to match what's in the paper; with the provided npz and prompts text text file, with PixArt-alpha 256x256, clean_fid with inception_v3 I get 24.85 FID. I also used batch size 100.

May I ask if you have tried the npz and prompt files we provided?

[mgwillia]

Yep, that's what I meant earlier.

[Shenyi-Z]

Consider using pytorch_fid to measure FID?

[mgwillia]

With pytorch_fid it's still quite different: 25.47.

Could you confirm the classifier-free guidance value you used for your FID>28 runs?

[Shenyi-Z]

Okay, I've been a bit busy lately, but I'll check the results here as soon as possible.