Reproduction results about FLUX

[leaves162]

Great work! While reproducing the results, I found that FLUX could not achieve the reported performance on DrawBench200 as stated in the paper (N=6, O=1, ImageReward=0.9953, CLIP=19.637). Since you didn’t provide the complete evaluation code, I’m not sure whether the issue lies in my own implementation.

I used the following command for generation:

CUDA_VISIBLE_DEVICES=0 python src/sample.py
--prompt_file DrawBench200.txt
--width 1024
--height 1024
--model_name flux-dev
--add_sampling_metadata
--output_dir ./results
--num_steps 50

I evaluated the generated images using ImageReward (ImageReward-v1.0 version) and CLIPScore (from torchmetrics.multimodal.clip_score, with the openai/clip-vit-large-patch14 version).
The final results I obtained were: ImageReward = 0.9410, CLIP = 17.1656, which show a significant gap compared to the reported results.
I'm not sure if there's anything wrong with this implementation or other factors I may have overlooked.

Looking forward to your response. Thank you again for your outstanding work!

[Shenyi-Z]

Thank you for your attention! This value does indeed show a significant deviation compared to the experimental error. A good troubleshooting method is to run the flux without acceleration in your current environment, setting N to 1. How are the ImageReward and CLIP scores in this case?

[leaves162]

Thank you very much for your feedback. I ran the same code with the configuration N=1, and the final results were ImageReward = 0.9649 and CLIP = 17.1667, which still show a significant gap compared to the originally reported results in the paper.

I implemented the evaluation code myself based on the settings provided in your paper. However, it’s possible that there are some flaws in my implementation. My code is as follows:

import os
import torch
from PIL import Image
from torchvision.transforms import ToTensor
from tqdm import tqdm
import argparse
import re
from torchmetrics.multimodal.clip_score import CLIPScore
import ImageReward as RM

with open(args.prompt_path, 'r', encoding='utf-8') as f:
        prompt_list = [line.strip() for line in f if line.strip()]
image_files = [
        f for f in os.listdir(args.generated_path)
        if f.lower().endswith(".jpg")
    ]
image_files = sorted(image_files, key=lambda x: int(re.search(r'\d+', x).group()))    

RMmodel = RM.load("ImageReward-v1.0").to(device)
total_score = 0.0
with torch.no_grad():
    for i in tqdm(range(0, len(image_files)), desc="Computing Image Reward"):
        score = RMmodel.score(prompt_list[i], os.path.join(args.generated_path, image_files[i]))
        total_score+=score
ir_score = total_score / len(prompt_list)
print(f"Final Average ImageReward Score over {len(prompt_list)} images: {ir_score:.4f}")

clip_score_metric = CLIPScore(model_name_or_path="openai/clip-vit-large-patch14").to(device)
total_score = 0.0
batch_size = 32
for i in tqdm(range(0, len(image_files), batch_size), desc="Computing CLIP Scores"):
    batch_files = image_files[i:i+batch_size]
    images = []
    captions = prompt_list[i:i+batch_size]
    for filename in batch_files:
        image_path = os.path.join(args.generated_path, filename)
        image = Image.open(image_path).convert("RGB")
        image_tensor = ToTensor()(image).unsqueeze(0)
        images.append(image_tensor)
    image_batch = torch.cat(images, dim=0).to(device)
    with torch.no_grad():
        score = clip_score_metric(image_batch, captions)
    batch_score = score.item()
    total_score += batch_score * len(images)
average_clip_score = total_score / len(prompt_list)
print(f"Final Average CLIP Score over {total_count} images: {average_clip_score:.4f}")

Due to the limited details in the paper, I'm not sure if there are any aspects I may have overlooked. Could you please let me know if there is anything wrong with my implementation?

I truly appreciate your timely response!

[Shenyi-Z]

From an implementation perspective, the code appears to be correct.
It is worth noting that our CLIP evaluation should be based on ViT-G, which is relatively larger. However, based on our recent understanding, CLIP score does not seem to accurately reflect image quality.

In our recent work, we also observed that due to differences in devices and environments, metrics relying on perceptual models, such as image reward and CLIP score, tend to exhibit instability in the scores they assign to the same method. Nevertheless, comparisons conducted under consistent device and environmental conditions still hold significant evaluative value.

We recommend that you primarily rely on the results from your own implementation in your paper and not worry too much about minor discrepancies. Although numerical variations may arise due to hardware differences, the overall assessment of the method's performance remains largely unaffected.

Additionally, based on our latest findings, pixel-wise metrics like PSNR and SSIM are both highly recognized and stable, particularly for acceleration-related tasks. Please refer to the latest version of our arXiv paper, where we have made corresponding updates.

In conclusion, we highly recommend that future researchers adopt metrics such as PSNR and SSIM. They are straightforward to use, widely accepted, and particularly persuasive in the field of acceleration. (So far, we have not encountered any reviewers questioning PSNR, hh)