The Evaluator Effect in Usability Tests

Niels Ebbe Jacobsen; Morten Hertzum; Bonnie E. John

The Evaluator Effect in Usability Tests

Niels Ebbe Jacobsen, Morten Hertzum, Bonnie E. John

Department of Computer Science

Abstract

Usability tests are applied in industry to evaluate systems and in research as a yardstick for other usability evaluation methods. However, one potential threat to the reliability of usability tests has been left unaddressed: the evaluator effect. In this study, four evaluators analyzed four videotaped usability test sessions. Only 20% of the 93 unique problems were detected by all four evaluators and 46% were detected by only a single evaluator. Severe problems were detected more often by all four evaluators (41%) and less often by only one evaluator (22%) but a substantial evaluator effect remained

Original language	English
Title of host publication	CHI'98 Conference Summary
Number of pages	2
Place of Publication	New York
Publisher	ACM Press
Publication date	1998
Pages	255-256
Publication status	Published - 1998

Cite this

@inproceedings{0c04cd58c7d74164bc0d41df1bc91242,

title = "The Evaluator Effect in Usability Tests",

abstract = "Usability tests are applied in industry to evaluate systems and in research as a yardstick for other usability evaluation methods. However, one potential threat to the reliability of usability tests has been left unaddressed: the evaluator effect. In this study, four evaluators analyzed four videotaped usability test sessions. Only 20% of the 93 unique problems were detected by all four evaluators and 46% were detected by only a single evaluator. Severe problems were detected more often by all four evaluators (41%) and less often by only one evaluator (22%) but a substantial evaluator effect remained",

author = "Jacobsen, {Niels Ebbe} and Morten Hertzum and John, {Bonnie E.}",

year = "1998",

language = "English",

pages = "255--256",

booktitle = "CHI'98 Conference Summary",

publisher = "ACM Press",

}

TY - GEN

T1 - The Evaluator Effect in Usability Tests

AU - Jacobsen, Niels Ebbe

AU - Hertzum, Morten

AU - John, Bonnie E.

PY - 1998

Y1 - 1998

N2 - Usability tests are applied in industry to evaluate systems and in research as a yardstick for other usability evaluation methods. However, one potential threat to the reliability of usability tests has been left unaddressed: the evaluator effect. In this study, four evaluators analyzed four videotaped usability test sessions. Only 20% of the 93 unique problems were detected by all four evaluators and 46% were detected by only a single evaluator. Severe problems were detected more often by all four evaluators (41%) and less often by only one evaluator (22%) but a substantial evaluator effect remained

AB - Usability tests are applied in industry to evaluate systems and in research as a yardstick for other usability evaluation methods. However, one potential threat to the reliability of usability tests has been left unaddressed: the evaluator effect. In this study, four evaluators analyzed four videotaped usability test sessions. Only 20% of the 93 unique problems were detected by all four evaluators and 46% were detected by only a single evaluator. Severe problems were detected more often by all four evaluators (41%) and less often by only one evaluator (22%) but a substantial evaluator effect remained

M3 - Article in proceedings

SP - 255

EP - 256

BT - CHI'98 Conference Summary

PB - ACM Press

CY - New York

ER -

The Evaluator Effect in Usability Tests

Abstract

Fingerprint

Cite this