Permutation tests for classification: Revisited

TY - CHAP

T1 - Permutation tests for classification: Revisited

AU - Ganz, Melanie

AU - Konukoglu, Ender

PY - 2017/7/14

Y1 - 2017/7/14

N2 - In recent years, the focus on validating the statistical methods used in the field of neuroimaging has increased. While several papers have already highlighted the importance of non-parametric methods and especially permutation testing for general linear models (GLMs), it seems like the importance of validating classification results other than through cross-validation has taken a back seat. But classification, especially binary classification, is one of the most common tools in neuroimaging. Often permutations are not performed using the argument that they are too computationally expensive, especially for trainingintensive classifier as e.g. neural networks. In the following we want to re-visit the use of permutation tests for validating cross-validation results statistically and employ recent approximate permutation methods that reduce the number of permutations that need to be performed. We evaluate the feasibility of using full as well as approximate permutation methods in the extreme cases of small and unbalanced data sets. Our results indicate the applicability of a tail and Gamma approximation to perform permutation testing for binary classification tasks.

AB - In recent years, the focus on validating the statistical methods used in the field of neuroimaging has increased. While several papers have already highlighted the importance of non-parametric methods and especially permutation testing for general linear models (GLMs), it seems like the importance of validating classification results other than through cross-validation has taken a back seat. But classification, especially binary classification, is one of the most common tools in neuroimaging. Often permutations are not performed using the argument that they are too computationally expensive, especially for trainingintensive classifier as e.g. neural networks. In the following we want to re-visit the use of permutation tests for validating cross-validation results statistically and employ recent approximate permutation methods that reduce the number of permutations that need to be performed. We evaluate the feasibility of using full as well as approximate permutation methods in the extreme cases of small and unbalanced data sets. Our results indicate the applicability of a tail and Gamma approximation to perform permutation testing for binary classification tasks.

UR - http://www.mendeley.com/research/permutation-tests-classification-revisited

U2 - 10.1109/prni.2017.7981495

DO - 10.1109/prni.2017.7981495

M3 - Book chapter

SN - 9781538631591

T3 - 2017 International Workshop on Pattern Recognition in Neuroimaging, PRNI 2017

BT - 2017 International Workshop on Pattern Recognition in Neuroimaging, PRNI 2017

PB - Institute of Electrical and Electronics Engineers Inc.

ER -