Question about MS MARCO dataset query count in Table 14

[xiaohu677]

Hi,

Thank you for your great work and for open-sourcing the code!

In issue #12 you mentioned:
"In our paper and code we used BEIR dataset. We used test split for NQ and HotpotQA, and train split for MS-MARCO."

I'm still a bit confused: Table 14 in the paper lists 6,980 queries for the MS MARCO training set, but the official MS MARCO train.tsv contains far more (~500k+ unique queries with relevance labels).

Could you please share the exact 6,980-query subset used in the paper (or the file/link/processing step that loads/selects these queries)?

Thank you so much for your time and for any clarification!

[sleeepeer]

Hi,

Thank you for pointing this out.

I manually loaded these datasets to have a check. I can see for NQ and HotpotQA, the numbers of queries are correct (3,452 and 7,405 respectively). However when I was loading MSMARCO, seems for every split we get 502,939.

I used the following code to load them:

from src.utils import load_beir_datasets
corpus, queries, qrels = load_beir_datasets("msmarco", "train")
len(queries)

This might be a minor flaw but we can ensure the data we provide under results/ are what we used to run experiments in our paper. We entered the statistics in Table 14 from https://github.com/beir-cellar/beir.

Additionally, we recently fixed a bug—please pull the latest code before running experiments.

Thank you again for your interest in our work and for bringing this issue to our attention.

[xiaohu677]

Hi,

Thank you for your previous reply!

Just one more quick question:
Is the file results/beir_results/msmarco-contriever.json the complete set of retrieval results for MS MARCO used in your experiments?
In other words, does it contain the retrieval outputs for all the queries that were used for MS MARCO in your paper/experiments?

Thanks again for your time and help!