The Rating Game: Sentiment Rating Reproducibility from Text

TY - GEN

T1 - The Rating Game: Sentiment Rating Reproducibility from Text

AU - Borgholm, Lasse

AU - Simonson, Peter

AU - Hovy, Dirk

PY - 2015

Y1 - 2015

N2 - Sentiment analysis models often use ratings as labels, assuming that these ratings reflect the sentiment of the accompanying text. We investigate (i) whether human readers can infer ratings from review text, (ii) how human performance compares to a regression model, and (iii) whether model performance is affected by the rating "source" (i.e. original author vs. annotator). We collect IMDb movie reviews with author-provided ratings, and have them re-annotated by crowdsourced and trained annotators. Annotators reproduce the original ratings better than a model, but are still far off in more than 5% of the cases. Models trained on annotator-labels outperform those trained on author-labels, questioning the usefulness of author-rated reviews as training data for sentiment analysis.

AB - Sentiment analysis models often use ratings as labels, assuming that these ratings reflect the sentiment of the accompanying text. We investigate (i) whether human readers can infer ratings from review text, (ii) how human performance compares to a regression model, and (iii) whether model performance is affected by the rating "source" (i.e. original author vs. annotator). We collect IMDb movie reviews with author-provided ratings, and have them re-annotated by crowdsourced and trained annotators. Annotators reproduce the original ratings better than a model, but are still far off in more than 5% of the cases. Models trained on annotator-labels outperform those trained on author-labels, questioning the usefulness of author-rated reviews as training data for sentiment analysis.

M3 - Article in proceedings

SN - 978-1-941643-32-7

SP - 2527

EP - 2532

BT - 2015 Conference on Empirical Methods for Natural Language Processing

PB - Association for Computational Linguistics

CY - Lisbon, Portugal

ER -