An empirical study of differences between conversion schemes and annotation guidelines

Anders Søgaard

An empirical study of differences between conversion schemes and annotation guidelines

Centre for Language Technology

Abstract

We establish quantitative methods for comparing and estimating the quality of dependency annotations or conversion schemes. We use generalized tree-edit distance to measure divergence between annotations and propose theoretical learnability, derivational perplexity and downstream performance for evaluation. We present systematic experiments with tree-to-dependency conversions of the Penn-III treebank, as well as observations from experiments using treebanks from multiple languages. Our most important observations are: (a) parser bias makes most parsers insensitive to non-local differences between annotations, but (b) choice of annotation nevertheless has significant impact on most downstream applications, and (c) while learnability does not correlate with downstream performance, learnable annotations will lead to more robust performance across domains.

Original language	English
Title of host publication	DepLing 2013 : Proceedings of the Second International Conference on Dependency Linguistics 2013
Number of pages	9
Place of Publication	Prag
Publisher	Association for Computational Linguistics
Publication date	2013
Pages	298-306
ISBN (Electronic)	978-80-7378-240-5
Publication status	Published - 2013
Event	International Conference on Dependency Linguistics: DepLing - Prag, Czech Republic Duration: 27 Aug 2013 → 30 Aug 2013 Conference number: 2

Conference

Conference	International Conference on Dependency Linguistics
Number	2
Country/Territory	Czech Republic
City	Prag
Period	27/08/2013 → 30/08/2013

Cite this

An empirical study of differences between conversion schemes and annotation guidelines. / Søgaard, Anders.

DepLing 2013: Proceedings of the Second International Conference on Dependency Linguistics 2013. Prag : Association for Computational Linguistics, 2013. p. 298-306.

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

@inproceedings{0cdc7bf7fe7d4266a26d2c036a504023,

title = "An empirical study of differences between conversion schemes and annotation guidelines",

abstract = "We establish quantitative methods for comparing and estimating the quality of dependency annotations or conversion schemes. We use generalized tree-edit distance to measure divergence between annotations and propose theoretical learnability, derivational perplexity and downstream performance for evaluation. We present systematic experiments with tree-to-dependency conversions of the Penn-III treebank, as well as observations from experiments using treebanks from multiple languages. Our most important observations are: (a) parser bias makes most parsers insensitive to non-local differences between annotations, but (b) choice of annotation nevertheless has significant impact on most downstream applications, and (c) while learnability does not correlate with downstream performance, learnable annotations will lead to more robust performance across domains.",

author = "Anders S{\o}gaard",

year = "2013",

language = "English",

pages = "298--306",

booktitle = "DepLing 2013",

publisher = "Association for Computational Linguistics",

note = "International Conference on Dependency Linguistics ; Conference date: 27-08-2013 Through 30-08-2013",

}

TY - GEN

T1 - An empirical study of differences between conversion schemes and annotation guidelines

AU - Søgaard, Anders

N1 - Conference code: 2

PY - 2013

Y1 - 2013

N2 - We establish quantitative methods for comparing and estimating the quality of dependency annotations or conversion schemes. We use generalized tree-edit distance to measure divergence between annotations and propose theoretical learnability, derivational perplexity and downstream performance for evaluation. We present systematic experiments with tree-to-dependency conversions of the Penn-III treebank, as well as observations from experiments using treebanks from multiple languages. Our most important observations are: (a) parser bias makes most parsers insensitive to non-local differences between annotations, but (b) choice of annotation nevertheless has significant impact on most downstream applications, and (c) while learnability does not correlate with downstream performance, learnable annotations will lead to more robust performance across domains.

AB - We establish quantitative methods for comparing and estimating the quality of dependency annotations or conversion schemes. We use generalized tree-edit distance to measure divergence between annotations and propose theoretical learnability, derivational perplexity and downstream performance for evaluation. We present systematic experiments with tree-to-dependency conversions of the Penn-III treebank, as well as observations from experiments using treebanks from multiple languages. Our most important observations are: (a) parser bias makes most parsers insensitive to non-local differences between annotations, but (b) choice of annotation nevertheless has significant impact on most downstream applications, and (c) while learnability does not correlate with downstream performance, learnable annotations will lead to more robust performance across domains.

M3 - Article in proceedings

SP - 298

EP - 306

BT - DepLing 2013

PB - Association for Computational Linguistics

CY - Prag

T2 - International Conference on Dependency Linguistics

Y2 - 27 August 2013 through 30 August 2013

ER -

An empirical study of differences between conversion schemes and annotation guidelines

Abstract

Conference

Fingerprint

Cite this