TY - JOUR
T1 - Assessing and quantifying inter-rater variation for dichotomous ratings using a Rasch model
AU - Petersen, Jørgen Holm
AU - Larsen, Klaus
AU - Kreiner, Svend
PY - 2012/12
Y1 - 2012/12
N2 - We present a new model-based approach to the analysis of agreement between raters in a situation where all raters have supplied dichotomous ratings of the same cases in a sample. The model is a logistic regression model with random effects - a Rasch model. In the rater setting, the Rasch model includes parameters that allow raters to have different propensities to score a given set of individuals positively or negatively - the rater bias. An exact score test of the hypothesis of no rater bias is proposed and is shown to be an exact generalised McNemar's test. Based on the model, we suggest quantifying the rater variation as a suitable measure of the variation of the rater odds ratios. An important example that will serve to motivate and illustrate the proposed model, is the study of Umbilical artery Doppler velocimetry used by obstetricians to assess the status of a foetus. The purpose of the assessment is to improve the foetus' chance of survival by choosing the optimal time of elective delivery. In the study, data related to 139 perinatal deaths were sent to 32 experts who were asked whether the use of Doppler velocimetry might have prevented each death.
AB - We present a new model-based approach to the analysis of agreement between raters in a situation where all raters have supplied dichotomous ratings of the same cases in a sample. The model is a logistic regression model with random effects - a Rasch model. In the rater setting, the Rasch model includes parameters that allow raters to have different propensities to score a given set of individuals positively or negatively - the rater bias. An exact score test of the hypothesis of no rater bias is proposed and is shown to be an exact generalised McNemar's test. Based on the model, we suggest quantifying the rater variation as a suitable measure of the variation of the rater odds ratios. An important example that will serve to motivate and illustrate the proposed model, is the study of Umbilical artery Doppler velocimetry used by obstetricians to assess the status of a foetus. The purpose of the assessment is to improve the foetus' chance of survival by choosing the optimal time of elective delivery. In the study, data related to 139 perinatal deaths were sent to 32 experts who were asked whether the use of Doppler velocimetry might have prevented each death.
U2 - 10.1177/0962280210394168
DO - 10.1177/0962280210394168
M3 - Journal article
C2 - 21177706
SN - 0962-2802
JO - Statistical Methods in Medical Research
JF - Statistical Methods in Medical Research
ER -