Rethinking how to extend average precision to graded relevance

Marco Ferrante; Nicola Ferro; Maria Maistro

doi:10.1007/978-3-319-11382-1_3

Rethinking how to extend average precision to graded relevance

Marco Ferrante, Nicola Ferro, Maria Maistro

2 Citationer (Scopus)

Abstract

We present two new measures of retrieval effectiveness, inspired by Graded Average Precision(GAP), which extends Average Precision(AP) to graded relevance judgements. Starting from the random choice of a user, we define Extended Graded Average Precision(xGAP) and Expected Graded Average Precision(eGAP), which are more accurate than GAP in the case of a small number of highly relevant documents with high probability to be considered relevant by the users. The proposed measures are then evaluated on TREC 10, TREC 14, and TREC 21 collections showing that they actually grasp a different angle from GAP and that they are robust when it comes to incomplete judgments and shallow pools.

Originalsprog	Engelsk
Titel	Information Access Evaluation : Multilinguality, Multimodality, and Interaction - 5th International Conference of the CLEF Initiative, CLEF 2014, Proceedings
Antal sider	12
Forlag	Springer Verlag,
Publikationsdato	1 jan. 2014
Sider	19-30
ISBN (Trykt)	9783319113814
DOI	https://doi.org/10.1007/978-3-319-11382-1_3
Status	Udgivet - 1 jan. 2014
Begivenhed	5th International Conference of the CLEF Initiative, CLEF 2014 - Sheffield, Storbritannien Varighed: 15 sep. 2014 → 18 sep. 2014

Konference

Konference	5th International Conference of the CLEF Initiative, CLEF 2014
Land/Område	Storbritannien
By	Sheffield
Periode	15/09/2014 → 18/09/2014

Navn	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Vol/bind	8685 LNCS
ISSN	0302-9743

Adgang til dokumentet

10.1007/978-3-319-11382-1_3

Andre filer og links

Link to publication in Scopus

Citationsformater

Rethinking how to extend average precision to graded relevance. / Ferrante, Marco; Ferro, Nicola; Maistro, Maria.
Information Access Evaluation: Multilinguality, Multimodality, and Interaction - 5th International Conference of the CLEF Initiative, CLEF 2014, Proceedings. Springer Verlag, 2014. s. 19-30 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Bind 8685 LNCS).

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › peer review

Ferrante, M, Ferro, N & Maistro, M 2014, Rethinking how to extend average precision to graded relevance. i Information Access Evaluation: Multilinguality, Multimodality, and Interaction - 5th International Conference of the CLEF Initiative, CLEF 2014, Proceedings. Springer Verlag, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), bind 8685 LNCS, s. 19-30, 5th International Conference of the CLEF Initiative, CLEF 2014, Sheffield, Storbritannien, 15/09/2014. https://doi.org/10.1007/978-3-319-11382-1_3

Ferrante M, Ferro N, Maistro M. Rethinking how to extend average precision to graded relevance. I Information Access Evaluation: Multilinguality, Multimodality, and Interaction - 5th International Conference of the CLEF Initiative, CLEF 2014, Proceedings. Springer Verlag,. 2014. s. 19-30. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Bind 8685 LNCS). doi: 10.1007/978-3-319-11382-1_3

Ferrante, Marco ; Ferro, Nicola ; Maistro, Maria. / Rethinking how to extend average precision to graded relevance. Information Access Evaluation: Multilinguality, Multimodality, and Interaction - 5th International Conference of the CLEF Initiative, CLEF 2014, Proceedings. Springer Verlag, 2014. s. 19-30 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Bind 8685 LNCS).

@inproceedings{6d4d669b45ac4d548b30e254d0f1d26b,

title = "Rethinking how to extend average precision to graded relevance",

abstract = "We present two new measures of retrieval effectiveness, inspired by Graded Average Precision(GAP), which extends Average Precision(AP) to graded relevance judgements. Starting from the random choice of a user, we define Extended Graded Average Precision(xGAP) and Expected Graded Average Precision(eGAP), which are more accurate than GAP in the case of a small number of highly relevant documents with high probability to be considered relevant by the users. The proposed measures are then evaluated on TREC 10, TREC 14, and TREC 21 collections showing that they actually grasp a different angle from GAP and that they are robust when it comes to incomplete judgments and shallow pools.",

author = "Marco Ferrante and Nicola Ferro and Maria Maistro",

year = "2014",

month = jan,

day = "1",

doi = "10.1007/978-3-319-11382-1_3",

language = "English",

isbn = "9783319113814",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag,",

pages = "19--30",

booktitle = "Information Access Evaluation",

note = "5th International Conference of the CLEF Initiative, CLEF 2014 ; Conference date: 15-09-2014 Through 18-09-2014",

}

TY - GEN

T1 - Rethinking how to extend average precision to graded relevance

AU - Ferrante, Marco

AU - Ferro, Nicola

AU - Maistro, Maria

PY - 2014/1/1

Y1 - 2014/1/1

N2 - We present two new measures of retrieval effectiveness, inspired by Graded Average Precision(GAP), which extends Average Precision(AP) to graded relevance judgements. Starting from the random choice of a user, we define Extended Graded Average Precision(xGAP) and Expected Graded Average Precision(eGAP), which are more accurate than GAP in the case of a small number of highly relevant documents with high probability to be considered relevant by the users. The proposed measures are then evaluated on TREC 10, TREC 14, and TREC 21 collections showing that they actually grasp a different angle from GAP and that they are robust when it comes to incomplete judgments and shallow pools.

AB - We present two new measures of retrieval effectiveness, inspired by Graded Average Precision(GAP), which extends Average Precision(AP) to graded relevance judgements. Starting from the random choice of a user, we define Extended Graded Average Precision(xGAP) and Expected Graded Average Precision(eGAP), which are more accurate than GAP in the case of a small number of highly relevant documents with high probability to be considered relevant by the users. The proposed measures are then evaluated on TREC 10, TREC 14, and TREC 21 collections showing that they actually grasp a different angle from GAP and that they are robust when it comes to incomplete judgments and shallow pools.

UR - http://www.scopus.com/inward/record.url?scp=84906733016&partnerID=8YFLogxK

U2 - 10.1007/978-3-319-11382-1_3

DO - 10.1007/978-3-319-11382-1_3

M3 - Article in proceedings

AN - SCOPUS:84906733016

SN - 9783319113814

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 19

EP - 30

BT - Information Access Evaluation

PB - Springer Verlag,

T2 - 5th International Conference of the CLEF Initiative, CLEF 2014

Y2 - 15 September 2014 through 18 September 2014

ER -

Rethinking how to extend average precision to graded relevance

Abstract

Konference

Adgang til dokumentet

Andre filer og links

Fingeraftryk

Citationsformater