What’s in a p-value in NLP?

TY - GEN

T1 - What’s in a p-value in NLP?

AU - Søgaard, Anders

AU - Johannsen, Anders Trærup

AU - Plank, Barbara

AU - Hovy, Dirk

AU - Martinez Alonso, Hector

PY - 2014

Y1 - 2014

N2 - In NLP, we need to document that our proposed methods perform significantly better with respect to standard metrics than previous approaches, typically by reporting p-values obtained by rank- or randomization-based tests. We show that significance results following current research standards are unreliable and, in addition, very sensitive to sample size, covariates such as sentence length, as well as to the existence of multiple metrics. We estimate that under the assumption of perfect metrics and unbiased data, we need a significance cut-off at ~0.0025 to reduce the risk of false positive results to <5%. Since in practice we often have considerable selection bias and poor metrics, this, however, will not do alone.

AB - In NLP, we need to document that our proposed methods perform significantly better with respect to standard metrics than previous approaches, typically by reporting p-values obtained by rank- or randomization-based tests. We show that significance results following current research standards are unreliable and, in addition, very sensitive to sample size, covariates such as sentence length, as well as to the existence of multiple metrics. We estimate that under the assumption of perfect metrics and unbiased data, we need a significance cut-off at ~0.0025 to reduce the risk of false positive results to <5%. Since in practice we often have considerable selection bias and poor metrics, this, however, will not do alone.

M3 - Article in proceedings

SP - 1

EP - 10

BT - Eighteenth Conference on Computational Natural Language Learning

PB - Association for Computational Linguistics

CY - Baltimore, Maryland, USA

ER -