Injecting user models and time into precision via Markov chains

TY - GEN

T1 - Injecting user models and time into precision via Markov chains

AU - Ferrante, Marco

AU - Ferro, Nicola

AU - Maistro, Maria

PY - 2014/1/1

Y1 - 2014/1/1

N2 - We propose a family of new evaluation measures, called Markov Precision (MP), which exploits continuous-time and discrete-time Markov chains in order to inject user models into precision. Continuous-time MP behaves like timecalibrated measures, bringing the time spent by the user into the evaluation of a system; discrete-time MP behaves like traditional evaluation measures. Being part of the same Markovian framework, the time-based and rank-based versions of MP produce values that are directly comparable. We show that it is possible to re-create average precision using specific user models and this helps in providing an explanation of Average Precision (AP) in terms of user models more realistic than the ones currently used to justify it. We also propose several alternative models that take into account different possible behaviors in scanning a ranked result list. Finally, we conduct a thorough experimental evaluation of MP on standard TREC collections in order to show that MP is as reliable as other measures and we provide an example of calibration of its time parameters based on click logs from Yandex.

AB - We propose a family of new evaluation measures, called Markov Precision (MP), which exploits continuous-time and discrete-time Markov chains in order to inject user models into precision. Continuous-time MP behaves like timecalibrated measures, bringing the time spent by the user into the evaluation of a system; discrete-time MP behaves like traditional evaluation measures. Being part of the same Markovian framework, the time-based and rank-based versions of MP produce values that are directly comparable. We show that it is possible to re-create average precision using specific user models and this helps in providing an explanation of Average Precision (AP) in terms of user models more realistic than the ones currently used to justify it. We also propose several alternative models that take into account different possible behaviors in scanning a ranked result list. Finally, we conduct a thorough experimental evaluation of MP on standard TREC collections in order to show that MP is as reliable as other measures and we provide an example of calibration of its time parameters based on click logs from Yandex.

KW - Evaluation

KW - Markov precision

KW - Time

KW - User model

UR - http://www.scopus.com/inward/record.url?scp=84904569307&partnerID=8YFLogxK

U2 - 10.1145/2600428.2609637

DO - 10.1145/2600428.2609637

M3 - Article in proceedings

AN - SCOPUS:84904569307

SN - 9781450322591

SP - 597

EP - 606

BT - SIGIR 2014 - Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval

PB - ASSOCIATION FOR COMPUTING MACHINERY. JOU

T2 - 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2014

Y2 - 6 July 2014 through 11 July 2014

ER -