Regarding the fine-tuning example for BEIR

[ikuyamada]

Hi,
Thank you for sharing this excellent work on pretraining retrieval models!

I noticed that this repository includes an example of fine-tuning the model for BEIR:
https://github.com/ma787639046/bowdpr/blob/main/examples/finetune/beir/README.md

If I understand correctly, the BEIR results reported in the paper do not involve fine-tuning on the MS MARCO dataset. Is my understanding correct? If so, are the experimental results of the fine-tuned model available anywhere?

I look forward to your reply!

[ma787639046]

Hi @ikuyamada, didn't get notified by your message. I'm sorry for the late reply.

Our BeIR results are fine-tuned only on MS-MARCO and evaluated on other BeIR datasets. This aligns with the idea of out-of-domain evaluation proposed in the BeIR paper. The fine-tuning script just follows the implementation of BeIR's train_msmarco_v3.py, cause it achieves slightly better downstream performances.

Sorry that I can't find my fine-tuned model for BeIR right now. If you want to reproduce the BeIR results, you may fine-tune on MS-MARCO with the above script.

If you have further questions, please leave a message for me.