Zero-shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens

Marek Rei; Anders Søgaard

Zero-shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens

16 Citations (Scopus)

Abstract

Can attention-or gradient-based visualization techniques be used to infer token-level labels for binary sequence tagging problems, using networks trained only on sentence-level labels? We construct a neural network architecture based on soft attention, train it as a binary sentence classifier and evaluate against tokenlevel annotation on four different datasets. Inferring token labels from a network provides a method for quantitatively evaluating what the model is learning, along with generating useful feedback in assistance systems. Our results indicate that attention-based methods are able to predict token-level labels more accurately, compared to gradient-based methods, sometimes even rivaling the supervised oracle network.

Original language	English
Title of host publication	Proceedings, 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies : (Long Ppaers)
Volume	1
Publisher	Association for Computational Linguistics
Publication date	2018
Pages	293–302
Publication status	Published - 2018
Event	16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - New Orleans, United States Duration: 1 Jun 2018 → 6 Jun 2018

Conference

Conference	16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Country/Territory	United States
City	New Orleans
Period	01/06/2018 → 06/06/2018

Cite this

Zero-shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens. / Rei, Marek; Søgaard, Anders.
Proceedings, 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies : (Long Ppaers). Vol. 1 Association for Computational Linguistics, 2018. p. 293–302.

Research output: Chapter in Book/Report/Conference proceeding › Article in proceedings › Research › peer-review

Rei, M & Søgaard, A 2018, Zero-shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens. in Proceedings, 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies : (Long Ppaers). vol. 1, Association for Computational Linguistics, pp. 293–302, 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, United States, 01/06/2018.

@inproceedings{b9ea626955e6424ab7abe7ed350417e2,

title = "Zero-shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens",

abstract = "Can attention-or gradient-based visualization techniques be used to infer token-level labels for binary sequence tagging problems, using networks trained only on sentence-level labels? We construct a neural network architecture based on soft attention, train it as a binary sentence classifier and evaluate against tokenlevel annotation on four different datasets. Inferring token labels from a network provides a method for quantitatively evaluating what the model is learning, along with generating useful feedback in assistance systems. Our results indicate that attention-based methods are able to predict token-level labels more accurately, compared to gradient-based methods, sometimes even rivaling the supervised oracle network.",

author = "Marek Rei and Anders S{\o}gaard",

year = "2018",

language = "English",

volume = "1",

pages = "293–302",

booktitle = "Proceedings, 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies",

publisher = "Association for Computational Linguistics",

note = "16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2018 ; Conference date: 01-06-2018 Through 06-06-2018",

}

TY - GEN

T1 - Zero-shot Sequence Labeling

T2 - 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

AU - Rei, Marek

AU - Søgaard, Anders

PY - 2018

Y1 - 2018

N2 - Can attention-or gradient-based visualization techniques be used to infer token-level labels for binary sequence tagging problems, using networks trained only on sentence-level labels? We construct a neural network architecture based on soft attention, train it as a binary sentence classifier and evaluate against tokenlevel annotation on four different datasets. Inferring token labels from a network provides a method for quantitatively evaluating what the model is learning, along with generating useful feedback in assistance systems. Our results indicate that attention-based methods are able to predict token-level labels more accurately, compared to gradient-based methods, sometimes even rivaling the supervised oracle network.

AB - Can attention-or gradient-based visualization techniques be used to infer token-level labels for binary sequence tagging problems, using networks trained only on sentence-level labels? We construct a neural network architecture based on soft attention, train it as a binary sentence classifier and evaluate against tokenlevel annotation on four different datasets. Inferring token labels from a network provides a method for quantitatively evaluating what the model is learning, along with generating useful feedback in assistance systems. Our results indicate that attention-based methods are able to predict token-level labels more accurately, compared to gradient-based methods, sometimes even rivaling the supervised oracle network.

M3 - Article in proceedings

VL - 1

SP - 293

EP - 302

BT - Proceedings, 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

PB - Association for Computational Linguistics

Y2 - 1 June 2018 through 6 June 2018

ER -

Zero-shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens

Abstract

Conference

Fingerprint

Cite this