Different results for Materzynska ("disentangling") dataset

[MassEast]

When reproducing Table 3 from https://arxiv.org/pdf/2304.04512, all went well until running python eval.py --dataset=disentangling -t, which gave me (after fixing the dataset usage) 77.78% and 82.78% instead of 43.27% resp. 71.93% (as written in Table 3).
What's wrong?
Thanks a lot for helping figure this out!

[azuma164]

@MassEast Hi! Thank you for your comment and PR!
Actually I do not remember well because I did this long ago...

Considering this is done two years ago, I suppose CLIP model has changed and improved because I did not specify CLIP model in the code base.

[MassEast]

Hey @azuma164, thanks a lot for the quick answer. So far I only looked at disentangling, paint, and your rta-100. Results exactly reproduced for the latter two, but not for disentangling. That's what got me wondering and that's why I suppose it's not about the (potentially) changing CLIP weights.