Tags: #notesonai Topics: [[Information Retrieval]] ID: 20230110184719 --- # Expected Reciprocal Rank Suppose for a query $q$, a system returns a ranked list of $K$ documents $d_{1}, \ldots, d_{K},$ where the probability that document $k$ satisfies the user query is given by the transform of the editorial grade assigned to the query-document pair, which we write $p\left(q, d_{k}\right)$. Then ERR is given by: $ERR =\sum_{k=1}^{K} \frac{1}{k} p\left(q, d_{k}\right) \prod_{i=1}^{k-1}\left(1-p\left(q, d_{i}\right)\right)$ Stopping at rank $k$ involves being satisfied with document $k$, the probability of which is $p\left(q, d_{k}\right)$. It also involves not having been satisified with any of the previous documents ranked $1, \ldots, k-1,$ the probability of which is $\prod_{i=1}^{k-1}\left(1-p\left(q, d_{i}\right)\right) .$ This is all multiplied by $1 / k$ because it's the inverse stopping rank whose expectation is being computed. We need to model how likely it is that a given document will satisfy a given user query i.e. $p\left(q, d_{k}\right)$. This is calculated as a function of the "editorial grade" of that document. Let $g_{i}$ be the grade of the $i$ -th document, then a "utility" function is defined as: $\mathcal{R}(g):=\frac{2^{g}-1}{2^{g_{\max }}}, \quad g \in\left\{0, \ldots, g_{\max }\right\}$ One nice feature of the expected reciprocal rank metric is that it always falls between 0 and 1, with 1 being best. This means that the loss due to messing up any single example is bounded. --- ## References 1. Chapelle, O., Metlzer, D., Zhang, Y., & Grinspan, P. (2009). _Expected reciprocal rank for graded relevance. Proceeding of the 18th ACM Conference on Information and Knowledge Management - CIKM ’09._ doi:10.1145/1645953.1646033 2. https://lingpipe-blog.com/2010/03/09/chapelle-metzler-zhang-grinspan-2009-expected-reciprocal-rank-for-graded-relevance/