Abstract
Using automatic measures such as labeled and unlabeled attachment scores is common practice in dependency parser evaluation. In this paper, we examine whether these measures correlate with human judgments of overall parse quality. We ask linguists with experience in dependency annotation to judge system outputs. We measure the correlation between their judgments and a range of parse evaluation metrics across five languages. The human-metric correlation is lower for dependency parsing than for other NLP tasks. Also, inter-annotator agreement is sometimes higher than the agreement between judgments and metrics, indicating that the standard metrics fail to capture certain aspects of parse quality, such as the relevance of root attachment or the relative importance of the different parts of speech.
| Originalsprog | Engelsk |
|---|---|
| Titel | The 19th Conference on Computational Natural Language Learning (CoNLL) |
| Antal sider | 5 |
| Forlag | Association for Computational Linguistics |
| Publikationsdato | 2015 |
| Sider | 315-320 |
| ISBN (Trykt) | 978-1-941643-77-8 |
| Status | Udgivet - 2015 |