Part of Speech Based Term Weighting for Information Retrieval

TY - GEN

T1 - Part of Speech Based Term Weighting for Information Retrieval

AU - Lioma, Christina

AU - Blanco, Roi

N1 - Published in: · Proceeding ECIR '09 Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval Pages 412 - 423 Springer-Verlag Berlin, Heidelberg ©2009 ISBN: 978-3-642-00957-0 doi>10.1007/978-3-642-00958-7_37

PY - 2009

Y1 - 2009

N2 - Automatic language processing tools typically assign to terms so-called `weights' corresponding to the contribution of terms to information content. Traditionally, term weights are computed from lexical statistics, e.g., term frequencies. We propose a new type of term weight that is computed from part of speech (POS) n-gram statistics. The proposed POS-based term weight represents how informative a term is in general, based on the `POS contexts' in which it generally occurs in language. We suggest five different computations of POS-based term weights by extending existing statistical approximations of term information measures. We apply these POS-based term weights to information retrieval, by integrating them into the model that matches documents to queries. Experiments with two TREC collections and 300 queries, using TF-IDF & BM25 as baselines, show that integrating our POS-based term weights to retrieval always leads to gains (up to +33.7% from the baseline). Additional experiments with a different retrieval model as baseline (Language Model with Dirichlet priors smoothing) and our best performing POS-based term weight, show retrieval gains always and consistently across the whole smoothing range of the baseline.

AB - Automatic language processing tools typically assign to terms so-called `weights' corresponding to the contribution of terms to information content. Traditionally, term weights are computed from lexical statistics, e.g., term frequencies. We propose a new type of term weight that is computed from part of speech (POS) n-gram statistics. The proposed POS-based term weight represents how informative a term is in general, based on the `POS contexts' in which it generally occurs in language. We suggest five different computations of POS-based term weights by extending existing statistical approximations of term information measures. We apply these POS-based term weights to information retrieval, by integrating them into the model that matches documents to queries. Experiments with two TREC collections and 300 queries, using TF-IDF & BM25 as baselines, show that integrating our POS-based term weights to retrieval always leads to gains (up to +33.7% from the baseline). Additional experiments with a different retrieval model as baseline (Language Model with Dirichlet priors smoothing) and our best performing POS-based term weight, show retrieval gains always and consistently across the whole smoothing range of the baseline.

M3 - Article in proceedings

SP - 412

EP - 423

BT - ECIR '09 Proceedings of the 31th European Conference on IR Research on Advances in Information Retrieval

ER -