Visual Representation for Visual Representation?

[HanaHangLee]

I don't see you using ResNet 101 and Fast RCNN as mentioned. Could you please provide me with clear information? I would really appreciate it.

[darkpromise98]

In this paper, we directly use the pre-processed region features for images. The region features are extracted by pre-trained ResNet-101 and Faster-RCNN models, which are proposed in the paper Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering . More information can be found in https://github.com/peteanderson80/bottom-up-attention.

[HanaHangLee]

I am fortunate to have the information you provided. I am a student working on a thesis on the topic of image-text matching. I just want to ask one more thing, could you give me an example of images passed into an FC (Fully Connected) layer in the form of BxNxD specifically? I really appreciate your help.

[darkpromise98]

Given a raw image with the shape [channel, height, weight] (e.g., the shape is [224, 224, 3] for RGB images), we have the three steps:

1. We use the dataloader to load the batch images, the batch size is B=128. The shape is [128, 224, 224, 3] / [batch_size, height, weight, channel].
2. We use the Faster-RCNN models to detect salient regions and extract the region features for each image, the feature dimension is D=2048, and the number of salient regions is N=36. Therefore, the shape of region features is [128, 36, 2048] / [B, N, D] for batch images.
3. We use the FC layer (without BatchNorm) to change feature dimensions, the input dimension is 2048, and the output dimension is 1024. Therefore, the FC layer makes a feature mapping: [128, 36, 2048] -> [128, 36, 1024].

[darkpromise98]

Besides, if you want to complete the image-text matching for your own image-text data, instead of our benchmark datasets (flickr30k and mscoco). You have to apply the pre-trained Faster-RCNN model ( https://github.com/peteanderson80/bottom-up-attention) to extract region features for raw images, since our framework is built on the top of pre-extracted region features.
Hence our framework is a two-stage framework: (1) extract the region features for raw images; (2) apply the extracted region features for our model.

If you want to find a one-stage/end-to-end framework, please follow our latest work in CVPR2024, the paper and codes will be released in June.

[HanaHangLee]

There are no words to express my gratitude clearly. I feel very excited to receive your explanations and suggestions, and I will definitely try my best to excel and be as dedicated as you are. I wish you every success in your upcoming endeavors.