CRA Metric Discrepancy in Style and Object Unlearning Tasks

[dokko1]

I'm currently reproducing the results from your paper using the official repository (UnlearnCanvas), but I'm encountering discrepancies in the CRA metric for both style and object unlearning tasks.

According to the paper, the reported CRA scores are:

• Style Unlearning: UA = 86.29, IRA = 84.59, CRA = 88.43
• Object Unlearning: UA = 75.43, IRA = 77.5, CRA = 81.18

However, in my reproduction using the provided code and instructions, I obtained the following results:

• Style Unlearning: UA = 89.21, IRA = 84.11, CRA = 82.54
• Object Unlearning: UA = 82.00, IRA = 75.49, CRA = 74.90

As you can see, while some metrics match or exceed your reported results, the CRA metric is consistently lower in both tasks.

It would be greatly appreciated if you could share any additional details about the experimental setup used in the paper—such as hyperparameters, seed settings, or evaluation code specifics—that might help explain this discrepancy.

[YtengTan]

Hi @dokko1,

Thank you for your interest in our work.

Our script src/clip/eval_uncanvas.py by default is searching unlearning step-size $\lambda$ over number of $3$ test-time generated images (see L163) and evaluate the unlearned model on a tiny set that has only $10$ generated images (see L374), for a quick demonstration purpose.

The discrepancy happens if you are running on this setup.

For Table 2 evaluations, we search $\lambda$ over number of $10$ test-time generated images (see Hyperparameters. page-6), and follow the evaluation setup in machine_unlearning/evaluation/quantitative/accuracy.py from UnlearnCanvas repository, where we take $80$ unlearned models, producing a complete answer set using seed $188, 288, 588, 688, 888$ (See L28) to evaluate the UA, IRA and CRA. For FID computation, we use the script machine_unlearning/evaluation/quantitative/fid.py from UnlearnCanvas.

I hope this response clarifies your concerns.