A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index

TY - JOUR

T1 - A note on the evaluation of novel biomarkers

T2 - do not rely on integrated discrimination improvement and net reclassification index

AU - Hilden, Jørgen

AU - Gerds, Thomas A

N1 - Copyright © 2013 John Wiley & Sons, Ltd.

PY - 2014/8/30

Y1 - 2014/8/30

N2 - The 'integrated discrimination improvement' (IDI) and the 'net reclassification index' (NRI) are statistics proposed as measures of the incremental prognostic impact that a new biomarker will have when added to an existing prediction model for a binary outcome. By design, both measures were meant to be intuitively appropriate, and the IDI and NRI formulae do look intuitively plausible. Both have become increasingly popular. We shall argue, however, that their use is not always safe. If IDI and NRI are used to measure gain in prediction performance, then poorly calibrated models may appear advantageous, and in a simulation study, even the model that actually generates the data (and hence is the best possible model) can be improved on without adding measured information. We illustrate these shortcomings in actual cancer data as well as by Monte Carlo simulations. In these examples, we contrast IDI and NRI with the area under ROC and the Brier score. Unlike IDI and NRI, these traditional measures have the characteristic that prognostic performance cannot be accidentally or deliberately inflated.

AB - The 'integrated discrimination improvement' (IDI) and the 'net reclassification index' (NRI) are statistics proposed as measures of the incremental prognostic impact that a new biomarker will have when added to an existing prediction model for a binary outcome. By design, both measures were meant to be intuitively appropriate, and the IDI and NRI formulae do look intuitively plausible. Both have become increasingly popular. We shall argue, however, that their use is not always safe. If IDI and NRI are used to measure gain in prediction performance, then poorly calibrated models may appear advantageous, and in a simulation study, even the model that actually generates the data (and hence is the best possible model) can be improved on without adding measured information. We illustrate these shortcomings in actual cancer data as well as by Monte Carlo simulations. In these examples, we contrast IDI and NRI with the area under ROC and the Brier score. Unlike IDI and NRI, these traditional measures have the characteristic that prognostic performance cannot be accidentally or deliberately inflated.

U2 - 10.1002/sim.5804

DO - 10.1002/sim.5804

M3 - Journal article

C2 - 23553436

SN - 0277-6715

VL - 33

SP - 3405

EP - 3414

JO - Statistics in Medicine

JF - Statistics in Medicine

IS - 19

ER -