User perspectives on query difficulty

Christina Lioma; Birger Larsen; Hinrich Schütze

doi:10.1007/978-3-642-23318-0_3

User perspectives on query difficulty

Christina Lioma, Birger Larsen, Hinrich Schütze

Department of Communication

2 Citations (Scopus)

Abstract

The difficulty of a user query can affect the performance of Information Retrieval (IR) systems. What makes a query difficult and how one may predict this is an active research area, focusing mainly on factors relating to the retrieval algorithm, to the properties of the retrieval data, or to statistical and linguistic features of the queries that may render them difficult. This work addresses query difficulty from a different angle, namely the users’ own perspectives on query difficulty. Two research questions are asked: (1) Are users aware that the query they submit to an IR system may be difficult for the system to address? (2) Are users aware of specific features in their query (e.g., domain-specificity, vagueness) that may render their query difficult for an IR system to address? A study of 420 queries from a Web search engine query log that are pre-categorised as easy, medium, hard by TREC based on system performance, reveals an interesting finding: users do not seem to reliably assess which query might be difficult; however, their assessments of which query features might render queries difficult are notably more accurate. Following this, a formal approach is presented for synthesising the user-assessed causes of query difficulty through opinion fusion into an overall assessment of query difficulty. The resulting assessments of query difficulty are found to agree notably more to the TREC categories than the direct user assessments.

Original language	English
Title of host publication	Advances in Information Retrieval Theory, Third International Conference, ICTIR 2011, Bertinoro, Italy, September 12-14, 2011
Editors	Gianni Amati, Fabio Crestani
Number of pages	12
Volume	6931
Place of Publication	Heidelberg
Publisher	Springer
Publication date	12 Sept 2011
Pages	3-14
DOIs	https://doi.org/10.1007/978-3-642-23318-0_3
Publication status	Published - 12 Sept 2011

Series	Lecture notes in computer science
Volume	6931

Access to Document

10.1007/978-3-642-23318-0_3

http://www.springerlink.com/content/055242814q11h840/

Cite this

User perspectives on query difficulty. / Lioma, Christina; Larsen, Birger; Schütze, Hinrich.

Advances in Information Retrieval Theory, Third International Conference, ICTIR 2011, Bertinoro, Italy, September 12-14, 2011. ed. / Gianni Amati; Fabio Crestani. Vol. 6931 Heidelberg : Springer, 2011. p. 3-14 (Lecture notes in computer science, Vol. 6931).

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Lioma, C, Larsen, B & Schütze, H 2011, User perspectives on query difficulty. in G Amati & F Crestani (eds), Advances in Information Retrieval Theory, Third International Conference, ICTIR 2011, Bertinoro, Italy, September 12-14, 2011. vol. 6931, Springer, Heidelberg, Lecture notes in computer science, vol. 6931, pp. 3-14. https://doi.org/10.1007/978-3-642-23318-0_3

@inproceedings{71ad1cb41f134be0bd5d8746b9493edb,

title = "User perspectives on query difficulty",

abstract = "The difficulty of a user query can affect the performance of Information Retrieval (IR) systems. What makes a query difficult and how one may predict this is an active research area, focusing mainly on factors relating to the retrieval algorithm, to the properties of the retrieval data, or to statistical and linguistic features of the queries that may render them difficult. This work addresses query difficulty from a different angle, namely the users{\textquoteright} own perspectives on query difficulty. Two research questions are asked: (1) Are users aware that the query they submit to an IR system may be difficult for the system to address? (2) Are users aware of specific features in their query (e.g., domain-specificity, vagueness) that may render their query difficult for an IR system to address? A study of 420 queries from a Web search engine query log that are pre-categorised as easy, medium, hard by TREC based on system performance, reveals an interesting finding: users do not seem to reliably assess which query might be difficult; however, their assessments of which query features might render queries difficult are notably more accurate. Following this, a formal approach is presented for synthesising the user-assessed causes of query difficulty through opinion fusion into an overall assessment of query difficulty. The resulting assessments of query difficulty are found to agree notably more to the TREC categories than the direct user assessments. ",

author = "Christina Lioma and Birger Larsen and Hinrich Sch{\"u}tze",

year = "2011",

month = sep,

day = "12",

doi = "10.1007/978-3-642-23318-0_3",

language = "English",

volume = "6931",

series = "Lecture notes in computer science",

publisher = "Springer",

pages = "3--14",

editor = "Gianni Amati and Fabio Crestani",

booktitle = "Advances in Information Retrieval Theory, Third International Conference, ICTIR 2011, Bertinoro, Italy, September 12-14, 2011",

}

TY - GEN

T1 - User perspectives on query difficulty

AU - Lioma, Christina

AU - Larsen, Birger

AU - Schütze, Hinrich

PY - 2011/9/12

Y1 - 2011/9/12

N2 - The difficulty of a user query can affect the performance of Information Retrieval (IR) systems. What makes a query difficult and how one may predict this is an active research area, focusing mainly on factors relating to the retrieval algorithm, to the properties of the retrieval data, or to statistical and linguistic features of the queries that may render them difficult. This work addresses query difficulty from a different angle, namely the users’ own perspectives on query difficulty. Two research questions are asked: (1) Are users aware that the query they submit to an IR system may be difficult for the system to address? (2) Are users aware of specific features in their query (e.g., domain-specificity, vagueness) that may render their query difficult for an IR system to address? A study of 420 queries from a Web search engine query log that are pre-categorised as easy, medium, hard by TREC based on system performance, reveals an interesting finding: users do not seem to reliably assess which query might be difficult; however, their assessments of which query features might render queries difficult are notably more accurate. Following this, a formal approach is presented for synthesising the user-assessed causes of query difficulty through opinion fusion into an overall assessment of query difficulty. The resulting assessments of query difficulty are found to agree notably more to the TREC categories than the direct user assessments.

AB - The difficulty of a user query can affect the performance of Information Retrieval (IR) systems. What makes a query difficult and how one may predict this is an active research area, focusing mainly on factors relating to the retrieval algorithm, to the properties of the retrieval data, or to statistical and linguistic features of the queries that may render them difficult. This work addresses query difficulty from a different angle, namely the users’ own perspectives on query difficulty. Two research questions are asked: (1) Are users aware that the query they submit to an IR system may be difficult for the system to address? (2) Are users aware of specific features in their query (e.g., domain-specificity, vagueness) that may render their query difficult for an IR system to address? A study of 420 queries from a Web search engine query log that are pre-categorised as easy, medium, hard by TREC based on system performance, reveals an interesting finding: users do not seem to reliably assess which query might be difficult; however, their assessments of which query features might render queries difficult are notably more accurate. Following this, a formal approach is presented for synthesising the user-assessed causes of query difficulty through opinion fusion into an overall assessment of query difficulty. The resulting assessments of query difficulty are found to agree notably more to the TREC categories than the direct user assessments.

U2 - 10.1007/978-3-642-23318-0_3

DO - 10.1007/978-3-642-23318-0_3

M3 - Article in proceedings

VL - 6931

T3 - Lecture notes in computer science

SP - 3

EP - 14

BT - Advances in Information Retrieval Theory, Third International Conference, ICTIR 2011, Bertinoro, Italy, September 12-14, 2011

A2 - Amati, Gianni

A2 - Crestani, Fabio

PB - Springer

CY - Heidelberg

ER -

User perspectives on query difficulty

Abstract

Access to Document

Fingerprint

Cite this