Mean Average Precision and Pooled Evaluations #36
Description
I came across an issue with the calculation of the mean average precision of pooled evaluations which results in mAP@k being lower than expected if k is lesser than the number of relevant documents in the pool.
Let r be the number of relevant documents in the pool, P(q,k) the precision of query q at cut-off k, d(q,i) the document at position i in query q and rel(q,d) the relevance of document d for query q with rel: Q × D -> {0,1}.
TrecTools calculates the average precision@k of a query q as
sum({P(q,i) ⋅ rel(q,d(q,i)) | i = {1,...,k}) / r
instead of
sum({P(q,i) ⋅ rel(q,d(q,i)) | i = {1,...,k}) / min(r,k)
resulting in counterintuitive results for AP (and, thus, mAP) if r < k.
I suggest adding a flag to get_map to set the denominator to min(r,k) instead of r for pooled query evaluations.
I've attached a minimal example with a query of ten documents, with the first five of them being relevant, and a total of ten relevant documents in the evaluation pool, for which TrecTools calculates the mAP@5 as 0.5 instead of 1.0.