The zero-shot performance of pre-trained ckpt is extremely low

[RenShuhuai-Andy]

Thanks for the great work.

I evaluate the zero-shot performance of the 25M pre-trained ckpt on the DiDeMo dataset, my command is

export VL_DATA_DIR=/home/renshuhuai/VindLU/
export VL_EXP_DIR=/home/renshuhuai/VindLU/output

dataset=didemo
pt_name=25M-pretrain.pth
ft_name=${pt_name}-ret_${dataset}

ngpus=4
num_frames=8
nfrm_test=8
batch_size=32

# evaluation
python tools/run.py --nnodes 1 --ngpus ${ngpus} --task retrieval \
    --jobname ${ft_name}/eval_${nfrm_test}frm --dep_jobname ${ft_name} \
    --config configs/ret_${dataset}.py \
    --no_slurm \
    --model_args "pretrained_path /home/renshuhuai/VindLU/checkpoints/${pt_name} \
        evaluate True test_types 'eval([\"test\"])'  num_frames_test ${nfrm_test}"

However, I got a extremely low result (e.g., 0.1 R@1 for video retrieval):

I want to know if this is normal?

[qyr0403]

have you solved this problem？i meet this too

[RenShuhuai-Andy]

not yet...

[qyr0403]

not yet...

but when i use 25M finetuned on msrvtt checkpoint to evaluate zero-shot performance on my own dataset, it works.
may be there is something wrong in ckpt's keys.