TY - JOUR
T1 - The Net Reclassification Index (NRI)
T2 - A Misleading Measure of Prediction Improvement Even with Independent Test Data Sets
AU - Pepe, Margaret S.
AU - Fan, Jing
AU - Feng, Ziding
AU - Gerds, Thomas
AU - Hilden, Jorgen
PY - 2015/10/1
Y1 - 2015/10/1
N2 - The Net Reclassification Index (NRI) is a very popular measure for evaluating the improvement in prediction performance gained by adding a marker to a set of baseline predictors. However, the statistical properties of this novel measure have not been explored in depth. We demonstrate the alarming result that the NRI statistic calculated on a large test dataset using risk models derived from a training set is likely to be positive even when the new marker has no predictive information. A related theoretical example is provided in which an incorrect risk function that includes an uninformative marker is proven to erroneously yield a positive NRI. Some insight into this phenomenon is provided. Since large values for the NRI statistic may simply be due to use of poorly fitting risk models, we suggest caution in using the NRI as the basis for marker evaluation. Other measures of prediction performance improvement, such as measures derived from the receiver operating characteristic curve, the net benefit function, and the Brier score, cannot be large due to poorly fitting risk functions.
AB - The Net Reclassification Index (NRI) is a very popular measure for evaluating the improvement in prediction performance gained by adding a marker to a set of baseline predictors. However, the statistical properties of this novel measure have not been explored in depth. We demonstrate the alarming result that the NRI statistic calculated on a large test dataset using risk models derived from a training set is likely to be positive even when the new marker has no predictive information. A related theoretical example is provided in which an incorrect risk function that includes an uninformative marker is proven to erroneously yield a positive NRI. Some insight into this phenomenon is provided. Since large values for the NRI statistic may simply be due to use of poorly fitting risk models, we suggest caution in using the NRI as the basis for marker evaluation. Other measures of prediction performance improvement, such as measures derived from the receiver operating characteristic curve, the net benefit function, and the Brier score, cannot be large due to poorly fitting risk functions.
U2 - 10.1007/s12561-014-9118-0
DO - 10.1007/s12561-014-9118-0
M3 - Journal article
C2 - 26504496
SN - 1867-1764
VL - 7
SP - 282
EP - 295
JO - Statistics in BioSciences
JF - Statistics in BioSciences
IS - 2
ER -