Mining until it hurts: automatic extraction of usability issues from online reviews compared to traditional usability evaluation

TY - GEN

T1 - Mining until it hurts

T2 - Nordic Conference on Human-Computer Interaction 2014

AU - Hedegaard, Steffen

AU - Simonsen, Jakob Grue

N1 - Conference code: 8

PY - 2014/10/26

Y1 - 2014/10/26

N2 - Large amounts of data available on the web, for example reviews, tweets, and forum postings, contain user narratives on interaction with products. Finding usability issues in such user narratives offers an interesting alternative to traditional usability testing. To leverage such data for identifying usability issues, we (I) devise a methodology for building automated extraction tools for usability issues; (II) perform empirical assessment of such tools by training a number of classifiers to extract sentences describing usability issues for two digital cameras and a children's tablet; (III) perform quantitative and qualitative comparisons between the usability issues identified by the classifiers and those identified and assessed by two traditional methods: heuristic evaluation and think aloud testing. Our results show that it is possible to build and train algorithms for extracting actionable usability issues, but raise serious concerns about the practical future prospects for supplementing traditional evaluation methods with automated extraction algorithms.

AB - Large amounts of data available on the web, for example reviews, tweets, and forum postings, contain user narratives on interaction with products. Finding usability issues in such user narratives offers an interesting alternative to traditional usability testing. To leverage such data for identifying usability issues, we (I) devise a methodology for building automated extraction tools for usability issues; (II) perform empirical assessment of such tools by training a number of classifiers to extract sentences describing usability issues for two digital cameras and a children's tablet; (III) perform quantitative and qualitative comparisons between the usability issues identified by the classifiers and those identified and assessed by two traditional methods: heuristic evaluation and think aloud testing. Our results show that it is possible to build and train algorithms for extracting actionable usability issues, but raise serious concerns about the practical future prospects for supplementing traditional evaluation methods with automated extraction algorithms.

KW - machine learning, natural language processing, usability, user experience

U2 - 10.1145/2639189.2639211

DO - 10.1145/2639189.2639211

M3 - Article in proceedings

SN - 978-1-4503-2542-4

SP - 157

EP - 166

BT - Proceedings of the 8th Nordic Conference on Human-Computer Interaction

PB - Association for Computing Machinery

Y2 - 26 October 2014 through 30 October 2014

ER -