An empirical study of differences between conversion schemes and annotation guidelines

Anders Søgaard

An empirical study of differences between conversion schemes and annotation guidelines

LUKKET: Center for Sprogteknologi

Abstract

We establish quantitative methods for comparing and estimating the quality of dependency annotations or conversion schemes. We use generalized tree-edit distance to measure divergence between annotations and propose theoretical learnability, derivational perplexity and downstream performance for evaluation. We present systematic experiments with tree-to-dependency conversions of the Penn-III treebank, as well as observations from experiments using treebanks from multiple languages. Our most important observations are: (a) parser bias makes most parsers insensitive to non-local differences between annotations, but (b) choice of annotation nevertheless has significant impact on most downstream applications, and (c) while learnability does not correlate with downstream performance, learnable annotations will lead to more robust performance across domains.

Originalsprog	Engelsk
Titel	DepLing 2013 : Proceedings of the Second International Conference on Dependency Linguistics 2013
Antal sider	9
Udgivelsessted	Prag
Forlag	Association for Computational Linguistics
Publikationsdato	2013
Sider	298-306
ISBN (Elektronisk)	978-80-7378-240-5
Status	Udgivet - 2013
Begivenhed	International Conference on Dependency Linguistics: DepLing - Prag, Tjekkiet Varighed: 27 aug. 2013 → 30 aug. 2013 Konferencens nummer: 2

Konference

Konference	International Conference on Dependency Linguistics
Nummer	2
Land/Område	Tjekkiet
By	Prag
Periode	27/08/2013 → 30/08/2013

Citationsformater

An empirical study of differences between conversion schemes and annotation guidelines. / Søgaard, Anders.

DepLing 2013: Proceedings of the Second International Conference on Dependency Linguistics 2013. Prag : Association for Computational Linguistics, 2013. s. 298-306.

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › peer review

@inproceedings{0cdc7bf7fe7d4266a26d2c036a504023,

title = "An empirical study of differences between conversion schemes and annotation guidelines",

abstract = "We establish quantitative methods for comparing and estimating the quality of dependency annotations or conversion schemes. We use generalized tree-edit distance to measure divergence between annotations and propose theoretical learnability, derivational perplexity and downstream performance for evaluation. We present systematic experiments with tree-to-dependency conversions of the Penn-III treebank, as well as observations from experiments using treebanks from multiple languages. Our most important observations are: (a) parser bias makes most parsers insensitive to non-local differences between annotations, but (b) choice of annotation nevertheless has significant impact on most downstream applications, and (c) while learnability does not correlate with downstream performance, learnable annotations will lead to more robust performance across domains.",

author = "Anders S{\o}gaard",

year = "2013",

language = "English",

pages = "298--306",

booktitle = "DepLing 2013",

publisher = "Association for Computational Linguistics",

note = "International Conference on Dependency Linguistics ; Conference date: 27-08-2013 Through 30-08-2013",

}

TY - GEN

T1 - An empirical study of differences between conversion schemes and annotation guidelines

AU - Søgaard, Anders

N1 - Conference code: 2

PY - 2013

Y1 - 2013

N2 - We establish quantitative methods for comparing and estimating the quality of dependency annotations or conversion schemes. We use generalized tree-edit distance to measure divergence between annotations and propose theoretical learnability, derivational perplexity and downstream performance for evaluation. We present systematic experiments with tree-to-dependency conversions of the Penn-III treebank, as well as observations from experiments using treebanks from multiple languages. Our most important observations are: (a) parser bias makes most parsers insensitive to non-local differences between annotations, but (b) choice of annotation nevertheless has significant impact on most downstream applications, and (c) while learnability does not correlate with downstream performance, learnable annotations will lead to more robust performance across domains.

AB - We establish quantitative methods for comparing and estimating the quality of dependency annotations or conversion schemes. We use generalized tree-edit distance to measure divergence between annotations and propose theoretical learnability, derivational perplexity and downstream performance for evaluation. We present systematic experiments with tree-to-dependency conversions of the Penn-III treebank, as well as observations from experiments using treebanks from multiple languages. Our most important observations are: (a) parser bias makes most parsers insensitive to non-local differences between annotations, but (b) choice of annotation nevertheless has significant impact on most downstream applications, and (c) while learnability does not correlate with downstream performance, learnable annotations will lead to more robust performance across domains.

M3 - Article in proceedings

SP - 298

EP - 306

BT - DepLing 2013

PB - Association for Computational Linguistics

CY - Prag

T2 - International Conference on Dependency Linguistics

Y2 - 27 August 2013 through 30 August 2013

ER -

An empirical study of differences between conversion schemes and annotation guidelines

Abstract

Konference

Fingeraftryk

Citationsformater