Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors

Asbjørn Hróbjartsson; Ann Sofia Skou Thomsen; Frida Emanuelsson; Britta Tendal; Jeppe Vejlgaard Rasmussen; Jørgen Hilden; Isabelle Boutron; Philippe Ravaud; Stig Brorson

doi:10.1093/ije/dyt270

Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors

Asbjørn Hróbjartsson, Ann Sofia Skou Thomsen, Frida Emanuelsson, Britta Tendal, Jeppe Vejlgaard Rasmussen, Jørgen Hilden, Isabelle Boutron, Philippe Ravaud, Stig Brorson

61 Citations (Scopus)

Abstract

Background: We wanted to evaluate the impact of nonblinded outcome assessors on estimated treatment effects in time-to-event trials. Methods: Systematic review of randomized clinical trials with both blinded and nonblinded assessors of the same time-to-event outcome. Two authors agreed on inclusion of trials and outcomes. We compared hazard ratios based on nonblinded and blinded assessments. A ratio of hazard ratios (RHR) <1 indicated that nonblinded assessors generated more optimistic effect estimates. We pooled RHRs with inverse variance random-effects meta-analysis. Results: We included 18 trials. Eleven trials (1969 patients) with subjective outcomes provided hazard ratios, RHR 0.88 (0.69 to 1.12), (I² = 44%, P = 0.06), but unconditional pooling was problematic because of qualitative heterogeneity. Four atypical cytomegalovirus retinitis trials compared experimental oral administration with control intravenous administration of the same drug, resulting in bias favouring the control intervention, RHR 1.33 (0.98 to 1.82). Seven trials of cytomegalovirus retinitis, tibial fracture and multiple sclerosis compared experimental interventions with standard control interventions, e.g. placebo, no-treatment or active control, resulting in bias favouring the experimental intervention, RHR 0.73 (0.57 to 0.93), indicating an average exaggeration of nonblinded hazard ratios by 27% (7% to 43%). Conclusions: Lack of blinded outcome assessors in randomized trials with subjective time-to-event outcomes causes high risk of observer bias. Nonblinded outcome assessors typically favour the experimental intervention, exaggerating the hazard ratio by an average of approximately 27%; but in special situations, nonblinded outcome assessors favour control interventions, inducing a comparable degree of observer bias in the reversed direction.

Original language	English
Journal	International Journal of Epidemiology
Volume	43
Issue number	3
Pages (from-to)	937-948
Number of pages	12
ISSN	0300-5771
DOIs	https://doi.org/10.1093/ije/dyt270
Publication status	Published - Jun 2014

Keywords

Humans
Observer Variation
Randomized Controlled Trials as Topic
Research Design
Single-Blind Method

Access to Document

10.1093/ije/dyt270

Cite this

Hróbjartsson, A., Thomsen, A. S. S., Emanuelsson, F., Tendal, B., Rasmussen, J. V., Hilden, J., Boutron, I., Ravaud, P., & Brorson, S. (2014). Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. International Journal of Epidemiology, 43(3), 937-948. https://doi.org/10.1093/ije/dyt270

Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors. / Hróbjartsson, Asbjørn; Thomsen, Ann Sofia Skou; Emanuelsson, Frida et al.
In: International Journal of Epidemiology, Vol. 43, No. 3, 06.2014, p. 937-948.

Research output: Contribution to journal › Journal article › Research › peer-review

Hróbjartsson, A, Thomsen, ASS, Emanuelsson, F, Tendal, B, Rasmussen, JV, Hilden, J, Boutron, I, Ravaud, P & Brorson, S 2014, 'Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors', International Journal of Epidemiology, vol. 43, no. 3, pp. 937-948. https://doi.org/10.1093/ije/dyt270

@article{2bc64b85ff5d4b62b5af0f29478d185b,

title = "Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors",

abstract = "Background: We wanted to evaluate the impact of nonblinded outcome assessors on estimated treatment effects in time-to-event trials. Methods: Systematic review of randomized clinical trials with both blinded and nonblinded assessors of the same time-to-event outcome. Two authors agreed on inclusion of trials and outcomes. We compared hazard ratios based on nonblinded and blinded assessments. A ratio of hazard ratios (RHR) <1 indicated that nonblinded assessors generated more optimistic effect estimates. We pooled RHRs with inverse variance random-effects meta-analysis. Results: We included 18 trials. Eleven trials (1969 patients) with subjective outcomes provided hazard ratios, RHR 0.88 (0.69 to 1.12), (I2 = 44%, P = 0.06), but unconditional pooling was problematic because of qualitative heterogeneity. Four atypical cytomegalovirus retinitis trials compared experimental oral administration with control intravenous administration of the same drug, resulting in bias favouring the control intervention, RHR 1.33 (0.98 to 1.82). Seven trials of cytomegalovirus retinitis, tibial fracture and multiple sclerosis compared experimental interventions with standard control interventions, e.g. placebo, no-treatment or active control, resulting in bias favouring the experimental intervention, RHR 0.73 (0.57 to 0.93), indicating an average exaggeration of nonblinded hazard ratios by 27% (7% to 43%). Conclusions: Lack of blinded outcome assessors in randomized trials with subjective time-to-event outcomes causes high risk of observer bias. Nonblinded outcome assessors typically favour the experimental intervention, exaggerating the hazard ratio by an average of approximately 27%; but in special situations, nonblinded outcome assessors favour control interventions, inducing a comparable degree of observer bias in the reversed direction.",

keywords = "Humans, Observer Variation, Randomized Controlled Trials as Topic, Research Design, Single-Blind Method",

author = "Asbj{\o}rn Hr{\'o}bjartsson and Thomsen, {Ann Sofia Skou} and Frida Emanuelsson and Britta Tendal and Rasmussen, {Jeppe Vejlgaard} and J{\o}rgen Hilden and Isabelle Boutron and Philippe Ravaud and Stig Brorson",

year = "2014",

month = jun,

doi = "10.1093/ije/dyt270",

language = "English",

volume = "43",

pages = "937--948",

journal = "International Journal of Epidemiology",

issn = "0300-5771",

publisher = "Oxford University Press",

number = "3",

}

TY - JOUR

T1 - Observer bias in randomized clinical trials with time-to-event outcomes

T2 - systematic review of trials with both blinded and non-blinded outcome assessors

AU - Hróbjartsson, Asbjørn

AU - Thomsen, Ann Sofia Skou

AU - Emanuelsson, Frida

AU - Tendal, Britta

AU - Rasmussen, Jeppe Vejlgaard

AU - Hilden, Jørgen

AU - Boutron, Isabelle

AU - Ravaud, Philippe

AU - Brorson, Stig

PY - 2014/6

Y1 - 2014/6

N2 - Background: We wanted to evaluate the impact of nonblinded outcome assessors on estimated treatment effects in time-to-event trials. Methods: Systematic review of randomized clinical trials with both blinded and nonblinded assessors of the same time-to-event outcome. Two authors agreed on inclusion of trials and outcomes. We compared hazard ratios based on nonblinded and blinded assessments. A ratio of hazard ratios (RHR) <1 indicated that nonblinded assessors generated more optimistic effect estimates. We pooled RHRs with inverse variance random-effects meta-analysis. Results: We included 18 trials. Eleven trials (1969 patients) with subjective outcomes provided hazard ratios, RHR 0.88 (0.69 to 1.12), (I2 = 44%, P = 0.06), but unconditional pooling was problematic because of qualitative heterogeneity. Four atypical cytomegalovirus retinitis trials compared experimental oral administration with control intravenous administration of the same drug, resulting in bias favouring the control intervention, RHR 1.33 (0.98 to 1.82). Seven trials of cytomegalovirus retinitis, tibial fracture and multiple sclerosis compared experimental interventions with standard control interventions, e.g. placebo, no-treatment or active control, resulting in bias favouring the experimental intervention, RHR 0.73 (0.57 to 0.93), indicating an average exaggeration of nonblinded hazard ratios by 27% (7% to 43%). Conclusions: Lack of blinded outcome assessors in randomized trials with subjective time-to-event outcomes causes high risk of observer bias. Nonblinded outcome assessors typically favour the experimental intervention, exaggerating the hazard ratio by an average of approximately 27%; but in special situations, nonblinded outcome assessors favour control interventions, inducing a comparable degree of observer bias in the reversed direction.

AB - Background: We wanted to evaluate the impact of nonblinded outcome assessors on estimated treatment effects in time-to-event trials. Methods: Systematic review of randomized clinical trials with both blinded and nonblinded assessors of the same time-to-event outcome. Two authors agreed on inclusion of trials and outcomes. We compared hazard ratios based on nonblinded and blinded assessments. A ratio of hazard ratios (RHR) <1 indicated that nonblinded assessors generated more optimistic effect estimates. We pooled RHRs with inverse variance random-effects meta-analysis. Results: We included 18 trials. Eleven trials (1969 patients) with subjective outcomes provided hazard ratios, RHR 0.88 (0.69 to 1.12), (I2 = 44%, P = 0.06), but unconditional pooling was problematic because of qualitative heterogeneity. Four atypical cytomegalovirus retinitis trials compared experimental oral administration with control intravenous administration of the same drug, resulting in bias favouring the control intervention, RHR 1.33 (0.98 to 1.82). Seven trials of cytomegalovirus retinitis, tibial fracture and multiple sclerosis compared experimental interventions with standard control interventions, e.g. placebo, no-treatment or active control, resulting in bias favouring the experimental intervention, RHR 0.73 (0.57 to 0.93), indicating an average exaggeration of nonblinded hazard ratios by 27% (7% to 43%). Conclusions: Lack of blinded outcome assessors in randomized trials with subjective time-to-event outcomes causes high risk of observer bias. Nonblinded outcome assessors typically favour the experimental intervention, exaggerating the hazard ratio by an average of approximately 27%; but in special situations, nonblinded outcome assessors favour control interventions, inducing a comparable degree of observer bias in the reversed direction.

KW - Humans

KW - Observer Variation

KW - Randomized Controlled Trials as Topic

KW - Research Design

KW - Single-Blind Method

U2 - 10.1093/ije/dyt270

DO - 10.1093/ije/dyt270

M3 - Journal article

C2 - 24448109

SN - 0300-5771

VL - 43

SP - 937

EP - 948

JO - International Journal of Epidemiology

JF - International Journal of Epidemiology

IS - 3

ER -

Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors

Abstract

Keywords

Access to Document

Fingerprint

Cite this