Towards a formal framework for utility-oriented measurements of retrieval effectiveness

Marco Ferrante; Nicola Ferro; Maria Maistro

doi:10.1145/2808194.2809452

Towards a formal framework for utility-oriented measurements of retrieval effectiveness

Marco Ferrante, Nicola Ferro, Maria Maistro

18 Citations (Scopus)

Abstract

In this paper we present a formal framework to define and study the properties of utility-oriented measurements of retrieval effectiveness, like AP, RBP, ERR and many other popular IR evaluation measures. The proposed framework is laid in the wake of the representational theory of measurement, which provides the foundations of the modern theory of measurement in both physical and social sciences, thus contributing to explicitly link IR evaluation to a broader context. The proposed framework is minimal, in the sense that it relies on just one axiom, from which other properties are derived. Finally, it contributes to a better understanding and a clear separation of what issues are due to the inherent problems in comparing systems in terms of retrieval effectiveness and what others are due to the expected numerical properties of a measurement.

Original language	English
Title of host publication	ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval
Number of pages	10
Publisher	Association for Computing Machinery, Inc.
Publication date	27 Sept 2015
Pages	21-30
ISBN (Electronic)	9781450338332
DOIs	https://doi.org/10.1145/2808194.2809452
Publication status	Published - 27 Sept 2015
Event	5th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2015 - Northampton, United States Duration: 27 Sept 2015 → 30 Sept 2015

Conference

Conference	5th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2015
Country/Territory	United States
City	Northampton
Period	27/09/2015 → 30/09/2015
Sponsor	ACM Special Interest Group on Information Retrieval (SIGIR)

Keywords

Balancing index
Omomorphism
Replacement
Representational theory of measurement
Swap

Access to Document

10.1145/2808194.2809452

Cite this

Towards a formal framework for utility-oriented measurements of retrieval effectiveness. / Ferrante, Marco; Ferro, Nicola; Maistro, Maria.
ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval. Association for Computing Machinery, Inc., 2015. p. 21-30.

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Ferrante, M, Ferro, N & Maistro, M 2015, Towards a formal framework for utility-oriented measurements of retrieval effectiveness. in ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval. Association for Computing Machinery, Inc., pp. 21-30, 5th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2015, Northampton, United States, 27/09/2015. https://doi.org/10.1145/2808194.2809452

@inproceedings{d9e469b6099f4972ac15708ee965cd16,

title = "Towards a formal framework for utility-oriented measurements of retrieval effectiveness",

abstract = "In this paper we present a formal framework to define and study the properties of utility-oriented measurements of retrieval effectiveness, like AP, RBP, ERR and many other popular IR evaluation measures. The proposed framework is laid in the wake of the representational theory of measurement, which provides the foundations of the modern theory of measurement in both physical and social sciences, thus contributing to explicitly link IR evaluation to a broader context. The proposed framework is minimal, in the sense that it relies on just one axiom, from which other properties are derived. Finally, it contributes to a better understanding and a clear separation of what issues are due to the inherent problems in comparing systems in terms of retrieval effectiveness and what others are due to the expected numerical properties of a measurement.",

keywords = "Balancing index, Omomorphism, Replacement, Representational theory of measurement, Swap",

author = "Marco Ferrante and Nicola Ferro and Maria Maistro",

year = "2015",

month = sep,

day = "27",

doi = "10.1145/2808194.2809452",

language = "English",

pages = "21--30",

booktitle = "ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval",

publisher = "Association for Computing Machinery, Inc.",

note = "5th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2015 ; Conference date: 27-09-2015 Through 30-09-2015",

}

TY - GEN

T1 - Towards a formal framework for utility-oriented measurements of retrieval effectiveness

AU - Ferrante, Marco

AU - Ferro, Nicola

AU - Maistro, Maria

PY - 2015/9/27

Y1 - 2015/9/27

N2 - In this paper we present a formal framework to define and study the properties of utility-oriented measurements of retrieval effectiveness, like AP, RBP, ERR and many other popular IR evaluation measures. The proposed framework is laid in the wake of the representational theory of measurement, which provides the foundations of the modern theory of measurement in both physical and social sciences, thus contributing to explicitly link IR evaluation to a broader context. The proposed framework is minimal, in the sense that it relies on just one axiom, from which other properties are derived. Finally, it contributes to a better understanding and a clear separation of what issues are due to the inherent problems in comparing systems in terms of retrieval effectiveness and what others are due to the expected numerical properties of a measurement.

AB - In this paper we present a formal framework to define and study the properties of utility-oriented measurements of retrieval effectiveness, like AP, RBP, ERR and many other popular IR evaluation measures. The proposed framework is laid in the wake of the representational theory of measurement, which provides the foundations of the modern theory of measurement in both physical and social sciences, thus contributing to explicitly link IR evaluation to a broader context. The proposed framework is minimal, in the sense that it relies on just one axiom, from which other properties are derived. Finally, it contributes to a better understanding and a clear separation of what issues are due to the inherent problems in comparing systems in terms of retrieval effectiveness and what others are due to the expected numerical properties of a measurement.

KW - Balancing index

KW - Omomorphism

KW - Replacement

KW - Representational theory of measurement

KW - Swap

UR - http://www.scopus.com/inward/record.url?scp=84964389739&partnerID=8YFLogxK

U2 - 10.1145/2808194.2809452

DO - 10.1145/2808194.2809452

M3 - Article in proceedings

AN - SCOPUS:84964389739

SP - 21

EP - 30

BT - ICTIR 2015 - Proceedings of the 2015 ACM SIGIR International Conference on the Theory of Information Retrieval

PB - Association for Computing Machinery, Inc.

T2 - 5th ACM SIGIR International Conference on the Theory of Information Retrieval, ICTIR 2015

Y2 - 27 September 2015 through 30 September 2015

ER -

Towards a formal framework for utility-oriented measurements of retrieval effectiveness

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this