(Potentially) A very important bug in extract_features_from_gt.py

[ZhangHanbo]

Hi, 12-in-1 is a very interesting work based on vilbert.
However, I am confused about the extract_features_from_gt.py scipt.

In the README under data/ directory, you said: to extract data features, users should firstly transform all grounding truth as the following format:

{
{
'file_name': 'name_of_image_file',
'file_path': '<path_to_image_file_on_your_disk>',
'bbox': array([
[ x1, y1, width1, height1],
[ x2, y2, width2, height2],
...
]),
'num_box': 2
},
....
}

However, I notice that in the extract_features_from_gt.py script, you do not recover the xywh to xyxy format, which should cause wrong feature extraction.
I am not sure whether this is an elaborate design or a bug.

Further, if this is a bug, what about the features used in VILBERT and 12-in-1? Are they correctly extracted using the correct bounding boxes?

[ZhangHanbo]

Here is a PyCharm debuging example:

[BoxList(num_boxes=2, image_width=800, image_height=1216, mode=xyxy), BoxList(num_boxes=2, image_width=800, image_height=1216, mode=xyxy)]

boxes for the 1st BoxList:
tensor([[194.7166, 569.9810, 251.4660, 337.0980],
[405.7705, 497.2300, 152.9555, 402.5910]], device='cuda:0')

boxes for the 2nd BoxList:
tensor([[293.1331, 173.9830, 144.2919, 466.2790],
[196.6867, 451.8960, 354.0944, 341.3350]], device='cuda:0')

[juliuswang0728]

Saw someone did mention the same issue in this PR.

[FightingFighting]

@ZhangHanbo Hi, Do you know how to generate the format of all grounding truth that you mentioned above.

[enaserianhanzaei]

@ZhangHanbo

what I did was simply modifying the extract_features function and saving the ground truth as allimages.npy:

def extract_features(self):
    f_list = []
    image_dir = self.args.image_dir
    if os.path.isfile(image_dir):
        features, infos = self.get_detectron_features([image_dir])
        self._save_feature(image_dir, features[0], infos[0])
    else:
        files = glob.glob(os.path.join(image_dir, "*"))
        # files = sorted(files)
        # files = [files[i: i+1000] for i in range(0, len(files), 1000)][self.args.partition]
        for chunk in self._chunks(files, self.args.batch_size):
            try:
                features, infos = self.get_detectron_features(chunk)

                for idx, file_name in enumerate(chunk):
                    self._save_feature(file_name, features[idx], infos[idx])

                    file_base_name = os.path.basename(file_name)
                    file_base_name = file_base_name.split(".")[0]
                    f_list.append(
                        {
                            "file_name": file_base_name,
                            "file_path": file_name,
                            "bbox": infos[idx]['bbox'],
                            "num_box": infos[idx]['num_boxes']
                        }
                    )

            except BaseException:
                continue
        
        np.save(os.path.join(self.args.output_folder, 'allimages.npy'), f_list)