Cannot reproduce the FID 4.02 in the paper with the pretrained ckpt of JiT-B/32 and example script

[sunyinggilly]

I am currently evaluating the pretrained checkpoint of JiT-B/32 provided in this link. Using the example script, I obtained results of FID 4.098 and IS 272.43 with a seed of 0. However, the results reported in the paper are FID 4.02 and IS 271.0.

Could you kindly confirm if these differences are within the expected range?

[SherlockJane]

Hi, when I was inferring JiT-B/32, I found that the output quality was poor, which is very similar to the issue mentioned in #34. I would like to ask, on what type of machine did you perform the inference? Thank you very much.

[sunyinggilly]

I used 4*A800 with bs 256 for each device. And, I have tried several random seeds (0, 3, 42, 10), where the FIDs are all around 4.10

[SherlockJane]

Thank you so much. I will keep trying. I ran the inference on an H100 and got results with semantic errors. The command I used is the one provided:
torchrun --nproc_per_node=8 --nnodes=1 --node_rank=0 \ main_jit.py \ --model JiT-B/32 \ --img_size 512 --noise_scale 2.0 \ --gen_bsz 256 --num_images 50000 --cfg 3.0 (or 2.9 or 0.1) --interval_min 0.1 --interval_max 1.0 \ --output_dir ${CKPT_DIR} --resume ${CKPT_DIR} \ --data_path ${IMAGENET_PATH} --evaluate_gen

[LTH14]

Yes -- that range is within the random variations.

[SherlockJane]

I still obtained such results on the A800. Could you please align the inference environment? I sincerely look forward to your reply.

• beautifulsoup4 4.14.3
• Jinja2 3.1.6
• numpy 1.22.3
• pillow 12.0.0
• scipy 1.9.1
• setuptools 80.9.0
• tensorboard 2.10.0
• tensorboard-data-server 0.6.1
• tensorboard-plugin-wit 1.8.1
• torch 2.5.1+cu121
• torchvision 0.20.1

[LTH14]

I don't see a clear problem with your generated image. I would suggest you compute the FID first to see whether it aligns.

[sunyinggilly]

@SherlockJane Have you tried the experiments in H100?

[SherlockJane]

@sunyinggilly Yes, I conducted the experiment on the 8*H100 with the inference environment as follows:

albumentations           2.0.8
Jinja2                   3.1.6
**numpy                    2.2.6**
pillow                   12.0.0
scipy                    1.15.3
setuptools               80.9.0
**torch                    2.5.1+cu124
torch_fidelity           0.4.0b0    
torchvision              0.20.1**
nvidia-cublas-cu12       12.4.5.8
nvidia-cuda-cupti-cu12   12.4.127
nvidia-cuda-nvrtc-cu12   12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12        9.1.0.70
nvidia-cufft-cu12        11.2.1.3
nvidia-curand-cu12       10.3.5.147
nvidia-cusolver-cu12     11.6.1.9
nvidia-cusparse-cu12     12.3.1.170
nvidia-nccl-cu12         2.21.5
nvidia-nvjitlink-cu12    12.4.127
nvidia-nvtx-cu12         12.4.127

I still obtained such visualization results, among which the FID and IS calculation for 800 images :
FID: 214.2411, Inception Score: 1.2156

[LTH14]

You should compute the FID on 50000 images instead of 800. 800 images will give a very bad FID.

[SherlockJane]

@LTH14 After computing 50,000 images, the results are obtained: FID: 189.6692, Inception Score: 1.2361. These values are highly unusual.

[LTH14]

Are they 50000 images from the same class or 1000 classes?

[LTH14]

From my experience, the visualization results you sent seem fine -- these results should not result in >50 FID.

[SherlockJane]

Yes, by using different labels, I can obtain valid FID results: FID: 4.0928, Inception Score: 269.7611. Thank you for your answers. ^_^