On the need for and provision of an 'IDEAL' scholarly information retrieval test collection

TY - GEN

T1 - On the need for and provision of an 'IDEAL' scholarly information retrieval test collection

AU - Larsen, Birger

AU - Lioma, Christina

N1 - Conference code: 3

PY - 2016

Y1 - 2016

N2 - Much about the potential, strengths and challenges of applying bibliometric techniques to scholarly Information Retrieval (IR) can be learned from conducting user studies with scholars and scientists, e.g. when interacting with bibliometrically enhanced IR prototypes or operational systems. However, most of IR research is still carried out as laboratory studies using test collections. As there is a lack of appropriate scholarly test collections, we argue in this position paper that one or more well-crafted test collections with scholarly documents is needed in order to engage the IR community whole-heartedly in Bibliometric IR research, and to facilitate foundational and high-quality work in this area. Based on the experiences gained from creating the iSearch test collection (Lykke et al. 2010 - http://itlab.dbit.dk/∼isearch/) we reflect on the properties of an ideal scholarly IR test collection and then examine three possible ways of realising the creation of such a collection. The first considers the possibilities for basing the test collection on the ever more readily available open access collections of scholarly documents; the second examines the possibilities of setting up alliances with proprietary content producers or aggregators; and the third considers collaboration with major university libraries and their content.

AB - Much about the potential, strengths and challenges of applying bibliometric techniques to scholarly Information Retrieval (IR) can be learned from conducting user studies with scholars and scientists, e.g. when interacting with bibliometrically enhanced IR prototypes or operational systems. However, most of IR research is still carried out as laboratory studies using test collections. As there is a lack of appropriate scholarly test collections, we argue in this position paper that one or more well-crafted test collections with scholarly documents is needed in order to engage the IR community whole-heartedly in Bibliometric IR research, and to facilitate foundational and high-quality work in this area. Based on the experiences gained from creating the iSearch test collection (Lykke et al. 2010 - http://itlab.dbit.dk/∼isearch/) we reflect on the properties of an ideal scholarly IR test collection and then examine three possible ways of realising the creation of such a collection. The first considers the possibilities for basing the test collection on the ever more readily available open access collections of scholarly documents; the second examines the possibilities of setting up alliances with proprietary content producers or aggregators; and the third considers collaboration with major university libraries and their content.

M3 - Article in proceedings

T3 - CEUR Workshop Proceedings

SP - 73

EP - 81

BT - Proceedings of the Third Workshop on Bibliometric-enhanced Information Retrieval (BIR 2016), Padova, Italy, March 20, 2016

A2 - Mayr, Philipp

A2 - Frommholz, Ingo

A2 - Cabanac, Guillaume

PB - CEUR-WS.org

T2 - Third Workshop on Bibliometric-enhanced Information Retrieval

Y2 - 20 March 2016 through 20 March 2016

ER -