An improved multileaving algorithm for online ranker evaluation

TY - GEN

T1 - An improved multileaving algorithm for online ranker evaluation

AU - Brost, Brian

AU - Cox, Ingemar Johansson

AU - Seldin, Yevgeny

AU - Lioma, Christina

N1 - Conference code: 39

PY - 2016/7/7

Y1 - 2016/7/7

N2 - Online ranker evaluation is a key challenge in information retrieval. An important task in the online evaluation of rankers is using implicit user feedback for inferring preferences between rankers. Interleaving methods have been found to be efficient and sensitive, i.e. they can quickly detect even small differences in quality. It has recently been shown that multileaving methods exhibit similar sensitivity but can be more efficient than interleaving methods. This paper presents empirical results demonstrating that existing multileaving methods either do not scale well with the number of rankers, or, more problematically, can produce results which substantially differ from evaluation measures like NDCG. The latter problem is caused by the fact that they do not correctly account for the similarities that can occur between rankers being multileaved. We propose a new multileaving method for handling this problem and demonstrate that it substantially outperforms existing methods, in some cases reducing errors by as much as 50%.

AB - Online ranker evaluation is a key challenge in information retrieval. An important task in the online evaluation of rankers is using implicit user feedback for inferring preferences between rankers. Interleaving methods have been found to be efficient and sensitive, i.e. they can quickly detect even small differences in quality. It has recently been shown that multileaving methods exhibit similar sensitivity but can be more efficient than interleaving methods. This paper presents empirical results demonstrating that existing multileaving methods either do not scale well with the number of rankers, or, more problematically, can produce results which substantially differ from evaluation measures like NDCG. The latter problem is caused by the fact that they do not correctly account for the similarities that can occur between rankers being multileaved. We propose a new multileaving method for handling this problem and demonstrate that it substantially outperforms existing methods, in some cases reducing errors by as much as 50%.

U2 - 10.1145/2911451.2914706

DO - 10.1145/2911451.2914706

M3 - Article in proceedings

SN - 978-1-4503-4069-4

SP - 745

EP - 748

BT - Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval

PB - Association for Computing Machinery

T2 - International ACM SIGIR conference on Research and Development in Information Retrieval 2016

Y2 - 17 July 2016 through 21 July 2016

ER -