Non-compositional term dependence for information retrieval

TY - GEN

T1 - Non-compositional term dependence for information retrieval

AU - Lioma, Christina

AU - Simonsen, Jakob Grue

AU - Larsen, Birger

AU - Hansen, Niels Dalum

PY - 2015/8/9

Y1 - 2015/8/9

N2 - Modelling term dependence in IR aims to identify co-occurring terms that are too heavily dependent on each other to be treated as a bag of words, and to adapt the indexing and ranking accordingly. Dependent terms are predominantly identified using lexical frequency statistics, assuming that (a) if terms co-occur often enough in some corpus, they are semantically dependent; (b) the more often they co-occur, the more semantically dependent they are. This assumption is not always correct: the frequency of co-occurring terms can be separate from the strength of their semantic dependence. E.g. red tape might be overall less frequent than tape measure in some corpus, but this does not mean that red+tape are less dependent than tape+measure. This is especially the case for non-compositional phrases, i.e. phrases whose meaning cannot be composed from the individual meanings of their terms (such as the phrase red tape meaning bureaucracy). Motivated by this lack of distinction between the frequency and strength of term dependence in IR, we present a principled approach for handling term dependence in queries, using both lexical frequency and semantic evidence. We focus on non-compositional phrases, extending a recent unsupervised model for their detection [21] to IR. Our approach, integrated into ranking using Markov Random Fields [31], yields effectiveness gains over competitive TREC baselines, showing that there is still room for improvement in the very well-studied area of term dependence in IR.

AB - Modelling term dependence in IR aims to identify co-occurring terms that are too heavily dependent on each other to be treated as a bag of words, and to adapt the indexing and ranking accordingly. Dependent terms are predominantly identified using lexical frequency statistics, assuming that (a) if terms co-occur often enough in some corpus, they are semantically dependent; (b) the more often they co-occur, the more semantically dependent they are. This assumption is not always correct: the frequency of co-occurring terms can be separate from the strength of their semantic dependence. E.g. red tape might be overall less frequent than tape measure in some corpus, but this does not mean that red+tape are less dependent than tape+measure. This is especially the case for non-compositional phrases, i.e. phrases whose meaning cannot be composed from the individual meanings of their terms (such as the phrase red tape meaning bureaucracy). Motivated by this lack of distinction between the frequency and strength of term dependence in IR, we present a principled approach for handling term dependence in queries, using both lexical frequency and semantic evidence. We focus on non-compositional phrases, extending a recent unsupervised model for their detection [21] to IR. Our approach, integrated into ranking using Markov Random Fields [31], yields effectiveness gains over competitive TREC baselines, showing that there is still room for improvement in the very well-studied area of term dependence in IR.

U2 - 10.1145/2766462.2767717

DO - 10.1145/2766462.2767717

M3 - Article in proceedings

SP - 595

EP - 604

BT - SIGIR '15

PB - Association for Computing Machinery

T2 - International ACM SIGIR Conference on Research and Development in Information Retrieval

Y2 - 9 August 2015 through 13 August 2015

ER -