Mean reciprocal rank

From Wikipedia, the free encyclopedia

This article does not cite any references or sources. (June 2007)
Please help improve this article by adding citations to reliable sources. Unverifiable material may be challenged and removed.

Mean reciprocal rank is a statistic for evaluating any process that produces a list of possible responses to a query, ordered by probability of correctness. The reciprocal rank of a query response is the multiplicative inverse of the rank of the correct answer. The mean reciprocal rank is the average of the reciprocal ranks of results for a sample of queries.

[edit] Example

For example, suppose we have the following three sample queries for a system that tries to translate English words to their plurals. In each case, the system makes three guesses, with the first one being the one it thinks is most likely correct:

Query	Results	Correct response	Rank	Reciprocal rank
cat	catten, cati, cats	cats	3	1/3
torus	torii, tori, toruses	tori	2	1/2
virus	viruses, virii, viri	viruses	1	1

Given those three samples, we could calculate the mean reciprocal rank as (1/3 + 1/2 + 1)/3 = 11/18 or about 0.61.

This basic definition does not specify what to do if (1) none of the proposed results are correct (use reciprocal rank 0), or if (2) there are multiple correct answers in the list (consider using Mean Average Precision: MAP).

[edit] References

Voorhees, E.M. (1999). "The TREC-8 question answering track report". In Proceedings of the 8th Text Retrieval Conference, Gaithersburg, Maryland, USA, pp. 77-82.