About the performance reproduce

[lwmming]

Thank you for your wonderful work! I have a little puzzle about the original acc of the six video models. Specifically, I run reference_ucf101.py to test original acc and get the results, i.e., {"i3d_resnet101": 96.03960396039604, "i3d_resnet50": 100.0, "slowfast_resnet101": 97.02970297029702, "slowfast_resnet50": 97.02970297029702, "tpn_resnet101": 98.01980198019803, "tpn_resnet50": 100.0}. Shouldn't it be all 100% according the statement in paper (...,which is correctly classified by all models,...)?
Looking forward to your reply. Thank you ~

[zhipeng-wei]

Thanks!

I check code again and find that the class funcion "attack_ucf101" is modified for readability(

TT/dataset/ucf101.py

Line 56 in 63a1a9d

class attack_ucf101(data.Dataset):

). I guess it maybe cause the lower accuracy. The major difference between these two versions is the path of setting. Note that 'test01_setting.txt' is downloaded from UCF101. Degrading this function to the orginal version may solve your question.
Welcome any questions ~

The original code is below:
`
class attack_ucf101(data.Dataset):

def __init__(self, spatial_transform=None, temporal_transform=None,get_loader=get_default_video_loader):     
    setting='/DATACENTER/zhipeng/UCF101/UCF101-annotation/ucfTrainTestlist/test01_setting.txt'    
    self.clips = self._make_dataset(setting)       
    self.spatial_transform = spatial_transform    
    self.temporal_transform = temporal_transform    
    self.loader = get_loader()     
    with open('/DATACENTER/zhipeng/UCF101/all_model_prediction/used_idxs.pkl', 'rb') as ipt:    
        used_idxs = pkl.load(ipt)    
    self.new_clips = []   
    for i in used_idxs:   
        self.new_clips.append(self.clips[i])   
    self.clips = self.new_clips   
    print ('length', len(self.clips))
def __getitem__(self, index):
    directory, duration, target = self.clips[index]
    frame_indices = list(range(1, duration + 1))
    if self.temporal_transform is not None:
        frame_indices = self.temporal_transform(frame_indices)
    clip = self.loader(directory, frame_indices)    
    if self.spatial_transform is not None:
        self.spatial_transform.randomize_parameters()
        clip = [self.spatial_transform(img) for img in clip]
    clip = torch.stack(clip, 0).permute(1, 0, 2, 3)
    return clip, target
def _make_dataset(self, setting):
    if not os.path.exists(setting):
        raise(RuntimeError("Setting file %s doesn't exist. Check opt.train-list and opt.val-list. " % (setting)))
    clips = []
    with open(setting) as split_f:
        data = split_f.readlines()
        for line in data:
            line_info = line.split()
            # line format: video_path, video_duration, video_label
            if len(line_info) < 3:
                raise(RuntimeError('Video input format is not correct, missing one or more element. %s' % line))
            clip_path = line_info[0]
            duration = int(line_info[1])
            target = int(line_info[2])
            item = (clip_path, duration, target)
            clips.append(item)
    return clips
def __len__(self):
    return len(self.clips)

`

[lwmming]

Thank you for you reply~
But I still cannot reproduce the attack performance. I run "python attack_ucf101.py --gpu 0 --batch_size 1 --model slowfast_resnet101 --attack_method TemporalTranslation --step 10 --file_prefix yours --momentum --kernlen 15 --move_type adj --kernel_mode gaussian" and get the result as {"i3d_resnet101": 50.495049504950494, "i3d_resnet50": 43.56435643564357, "slowfast_resnet101": 12.871287128712872, "slowfast_resnet50": 44.554455445544555, "tpn_resnet101": 71.2871287128713, "tpn_resnet50": 68.31683168316832}. This seems inconsistent with TT(10) in Figure3(a)(ii) in the paper. For example, the ASR of tpn_resnet50 here is about 31.69%, which is much higher than the results shown in the curve in Figure3(a)(ii).
Could you help me? Thank you so much~

[zhipeng-wei]

The result you reported is the accuracy of video models. Figure 3 shows the ASR of attacks.
Note that: ASR = 100% - accuracy. Thus, the accuracy of tpn_resnet50 is 68.31%, thus its ASR is 100%-68.31% = 31.69%.

[lwmming]

Yes, but the ASR of tpn_resnet50 shown in Figure 3(a)(ii) is less than 20%, which is inconsistent with 31.69%.

[zhipeng-wei]

Oh, sorry.
Can you make sure the number of steps? The performance is likely the one-step attack.

[lwmming]

Yes, the number of steps is 10. Besides, I also reproduce the one-step attack, its ASR is 39.60%.

[zhipeng-wei]

I notice that you assign momentum=Ture.
Figure 3 shows the performance of TT without momentum.
Please try "python attack_ucf101.py --gpu 0 --batch_size 1 --model slowfast_resnet101 --attack_method TemporalTranslation --step 10 --file_prefix yours --kernlen 15 --move_type adj --kernel_mode gaussian"

[lwmming]

Just like you said, I re-run it without momentum. But it gives 8.91% in terms of ASR on tpn_resnet50, which is still inconsistent with Figure 3(a)(ii).

[zhipeng-wei]

Sorry for the bad reproducing. I have some questions that:

1. are the attacks, using NL-101 and TPN-101 as white-box models on UCF-101, consistent with Figure 3 (a)?
2. are the ASRs of attacking Kinetics-400 consistent with Figure 3(b)?

I will check my code again. It may take some days.
If you solve this problem, please let me know.
Thanks for your issue again.

[lwmming]

1.When using NL-101 and TPN-101 as white-box models on UCF-101, the results reproduced by me are {"i3d_resnet101": 16.831683168316832, "i3d_resnet50": 55.445544554455445, "slowfast_resnet101": 87.12871287128714, "slowfast_resnet50": 85.14851485148515, "tpn_resnet101": 94.05940594059406, "tpn_resnet50": 92.07920792079207} and {"i3d_resnet101": 83.16831683168317, "i3d_resnet50": 80.1980198019802, "slowfast_resnet101": 93.06930693069307, "slowfast_resnet50": 84.15841584158416, "tpn_resnet101": 29.702970297029704, "tpn_resnet50": 86.13861386138613}, respectively. These results seem slightly different from Figure 3(a).
2. I have not run the setting in Kinetics-400.

[zhipeng-wei]

It seems like that the iterative version of TT is inconsistent with Figure 3(a).
I will check it.

[lwmming]

Hello~ I notice that you only give the model config files (.yaml) for kinetics400. This may be the cause of performance inconsistence. Could you please release the config files for models trained on UCF101? Thank you~

[zhipeng-wei]

It is not the cause. Models of Kinetics and UCF-101 use the same config files.
I have found my original output file. The results of using i3d_resnet101 as the white-box model are shown below
{"i3d_resnet50": 63.366336633663366, "i3d_resnet101": 11.881188118811881, "slowfast_resnet50": 81.18811881188118, "slowfast_resnet101": 85.14851485148515, "tpn_resnet50": 87.12871287128714, "tpn_resnet101": 92.07920792079207}.
The results of using slowfast_resnet101as the white-box model are shown below
{"i3d_resnet50": 72.27722772277228, "i3d_resnet101": 74.25742574257426, "slowfast_resnet50": 69.3069306930693, "slowfast_resnet101": 18.81188118811881, "tpn_resnet50": 86.13861386138613, "tpn_resnet101": 92.07920792079207}
And I rerun this repository, whose result is same with yours.
I guess the generation of attacked datasets is error, since I run my original code directly with UCF101 image folders. Although the performance reproduce is inconsistent with our result, they both have a similar trend.
I will further check and correct them.

I am busy recently, sorry for the late responce.