Cannot achieve the performance in the paper

[WanFang13]

Dear authors,
Thank you for sharing the code. I recently run the code on CIFAR10 dataset without changing any settings in the code. I achieved only 57.3% accuracy (44.0, 48.0, 52.7, 49.9, 41.6, 59.9, 57.3 for each cycle), which is far from 80.9% accuracy reported in the paper.

It seems that the settings in the code are not the final version. It will be very kind of you if you could provide the settings.

Best Wishes~

[GWwangshuo]

Hi, @Winfrand . Please check the Normalization for the training samples and test samples. The current code seems inconsistent. Specifically, just modify the cifar_transformer in main.py (line 19) as below.

def cifar_transformer():
    return transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.5, 0.5, 0.5,],
                                std=[0.5, 0.5, 0.5]),
        ])

[sinhasam]

@GWwangshuo good catch, thank you! i will fix this in the repo.

[WanFang13]

@GWwangshuo Thank you very much! I will try this new setting.

[danishmsin]

Hello Authors,
I ran the scripts with the modification suggested by @GWwangshuo. I am still not able to achieve the performance mentioned in the paper for CIFAR10 dataset.
I ran multiple times for averaging out the results. I am able to achieve accuracy between 67%-70% for 40% split.
It seems there are still some settings which are not present in the current version.
If so, kindly provide them.

Also, @Winfrand How much accuracy did you get after this modification?

Thanks :)

[gudovskiy]

@Winfrand @danishmsin I didn't check CIFAR10: only MNIST/SVHN/Imagenet and couldn't get good results for VAAL as well