TY - BOOK
T1 - Does it matter where we test?
T2 - Online user studies in digital libraries in natural environments
AU - Greifeneder, Elke
PY - 2012
Y1 - 2012
N2 - User studies in digital libraries face two fundamental challenges. The first is the necessity of running more user studies in an online environment. Users can access digital library collections and services worldwide and the services should be usable at any time. This need for more online studies is coupled with a second need, a demand to test under realistic conditions outside of laboratories in users’ natural environment. Asynchronous remote usability tests are a methodological approach that might answer both needs: they allow participants to take part in a study at a time and place of their choice, often in the participants’ natural environment. Any chosen place, however, might be noisy. Distractions are ubiquitous in a user’s natural environment. An awareness of the potential influences of distractions on users’ behavior during test situations is of great importance, because the validity of a study depends on the quality of the data. If an instrument allows systematic mistakes in measurements because of distractions, the validity is at risk. This dissertation examined if distraction in the users’ natural environment produces a systematic mistake in digital library studies that take place at a time and location of participants’ choice. In order to investigate the existence of distractions during online user studies in digital libraries and to analyze the influence(s) of that distraction, a psychological experiment was set up. It ex¬amined completion time scores between participants in a laboratory and participants in their natural environment. The results of the experiment showed that participants were highly distracted and that participants in their natural environment needed more time to complete the same test. The setting did not affect successful task completions, the participants’ judgments of sites or their decision-making processes. This work can conclude that it does not matter where we test, but it matters what happens during the test.
AB - User studies in digital libraries face two fundamental challenges. The first is the necessity of running more user studies in an online environment. Users can access digital library collections and services worldwide and the services should be usable at any time. This need for more online studies is coupled with a second need, a demand to test under realistic conditions outside of laboratories in users’ natural environment. Asynchronous remote usability tests are a methodological approach that might answer both needs: they allow participants to take part in a study at a time and place of their choice, often in the participants’ natural environment. Any chosen place, however, might be noisy. Distractions are ubiquitous in a user’s natural environment. An awareness of the potential influences of distractions on users’ behavior during test situations is of great importance, because the validity of a study depends on the quality of the data. If an instrument allows systematic mistakes in measurements because of distractions, the validity is at risk. This dissertation examined if distraction in the users’ natural environment produces a systematic mistake in digital library studies that take place at a time and location of participants’ choice. In order to investigate the existence of distractions during online user studies in digital libraries and to analyze the influence(s) of that distraction, a psychological experiment was set up. It ex¬amined completion time scores between participants in a laboratory and participants in their natural environment. The results of the experiment showed that participants were highly distracted and that participants in their natural environment needed more time to complete the same test. The setting did not affect successful task completions, the participants’ judgments of sites or their decision-making processes. This work can conclude that it does not matter where we test, but it matters what happens during the test.
KW - User Behavior
KW - natural environment
KW - online research
KW - distraction
M3 - Ph.D. thesis
BT - Does it matter where we test?
ER -