Early experiences with crowdsourcing airway annotations in chest CT

Veronika Cheplygina; Adria Perez-Rovira; Wieying Kuo; Harm A. W. M. Tiddens; Marleen de Bruijne

doi:10.1007/978-3-319-46976-8_22

Early experiences with crowdsourcing airway annotations in chest CT

Veronika Cheplygina^*, Adria Perez-Rovira, Wieying Kuo, Harm A. W. M. Tiddens, Marleen de Bruijne

^*Corresponding author af dette arbejde

Datalogisk Institut

12 Citationer (Scopus)

Abstract

Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.

Originalsprog	Engelsk
Titel	Deep Learning and Data Labeling for Medical Applications : First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, October 21, 2016, Proceedings
Redaktører	Gustavo Carneiro, Diana Mateus, Loïc Peter, Andrew Bradley, João Manuel R. S. Tavares, Vasileios Belagiannis, João Paulo Papa, Jacinto C. Nascimento, Marco Loog, Zhi Lu, Jaime S. Cardoso, Juilen Cornebise
Antal sider	10
Forlag	Springer
Publikationsdato	2016
Sider	209-218
ISBN (Trykt)	978-3-319-46975-1
ISBN (Elektronisk)	978-3-319-46976-8
DOI	https://doi.org/10.1007/978-3-319-46976-8_22
Status	Udgivet - 2016
Begivenhed	1st International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis - Athen, Grækenland Varighed: 21 okt. 2016 → 21 okt. 2016 Konferencens nummer: 1

Konference

Konference	1st International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis
Nummer	1
Land/Område	Grækenland
By	Athen
Periode	21/10/2016 → 21/10/2016

Navn	Lecture notes in computer science
Vol/bind	10008
ISSN	0302-9743

Adgang til dokumentet

10.1007/978-3-319-46976-8_22

Citationsformater

Cheplygina, V., Perez-Rovira, A., Kuo, W., Tiddens, H. A. W. M., & de Bruijne, M. (2016). Early experiences with crowdsourcing airway annotations in chest CT. I G. Carneiro, D. Mateus, L. Peter, A. Bradley, J. M. R. S. Tavares, V. Belagiannis, J. P. Papa, J. C. Nascimento, M. Loog, Z. Lu, J. S. Cardoso, & J. Cornebise (red.), Deep Learning and Data Labeling for Medical Applications: First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, October 21, 2016, Proceedings (s. 209-218). Springer. Lecture notes in computer science Bind 10008 https://doi.org/10.1007/978-3-319-46976-8_22

Early experiences with crowdsourcing airway annotations in chest CT. / Cheplygina, Veronika; Perez-Rovira, Adria; Kuo, Wieying et al.

Deep Learning and Data Labeling for Medical Applications: First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, October 21, 2016, Proceedings. red. / Gustavo Carneiro; Diana Mateus; Loïc Peter; Andrew Bradley; João Manuel R. S. Tavares; Vasileios Belagiannis; João Paulo Papa; Jacinto C. Nascimento; Marco Loog; Zhi Lu; Jaime S. Cardoso; Juilen Cornebise. Springer, 2016. s. 209-218 (Lecture notes in computer science, Bind 10008).

Publikation: Bidrag til bog/antologi/rapport › Konferencebidrag i proceedings › Forskning › peer review

Cheplygina, V, Perez-Rovira, A, Kuo, W, Tiddens, HAWM & de Bruijne, M 2016, Early experiences with crowdsourcing airway annotations in chest CT. i G Carneiro, D Mateus, L Peter, A Bradley, JMRS Tavares, V Belagiannis, JP Papa, JC Nascimento, M Loog, Z Lu, JS Cardoso & J Cornebise (red), Deep Learning and Data Labeling for Medical Applications: First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, October 21, 2016, Proceedings. Springer, Lecture notes in computer science, bind 10008, s. 209-218, 1st International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis, Athen, Grækenland, 21/10/2016. https://doi.org/10.1007/978-3-319-46976-8_22

Cheplygina V, Perez-Rovira A, Kuo W, Tiddens HAWM, de Bruijne M. Early experiences with crowdsourcing airway annotations in chest CT. I Carneiro G, Mateus D, Peter L, Bradley A, Tavares JMRS, Belagiannis V, Papa JP, Nascimento JC, Loog M, Lu Z, Cardoso JS, Cornebise J, red., Deep Learning and Data Labeling for Medical Applications: First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, October 21, 2016, Proceedings. Springer. 2016. s. 209-218. (Lecture notes in computer science, Bind 10008). doi: 10.1007/978-3-319-46976-8_22

Cheplygina, Veronika ; Perez-Rovira, Adria ; Kuo, Wieying et al. / Early experiences with crowdsourcing airway annotations in chest CT. Deep Learning and Data Labeling for Medical Applications: First International Workshop, LABELS 2016, and Second International Workshop, DLMIA 2016, Held in Conjunction with MICCAI 2016, Athens, Greece, October 21, 2016, Proceedings. red. / Gustavo Carneiro ; Diana Mateus ; Loïc Peter ; Andrew Bradley ; João Manuel R. S. Tavares ; Vasileios Belagiannis ; João Paulo Papa ; Jacinto C. Nascimento ; Marco Loog ; Zhi Lu ; Jaime S. Cardoso ; Juilen Cornebise. Springer, 2016. s. 209-218 (Lecture notes in computer science, Bind 10008).

@inproceedings{9ae10e45d7bf4196b4fa98e8f1a3062f,

title = "Early experiences with crowdsourcing airway annotations in chest CT",

abstract = "Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.",

author = "Veronika Cheplygina and Adria Perez-Rovira and Wieying Kuo and Tiddens, {Harm A. W. M.} and {de Bruijne}, Marleen",

year = "2016",

doi = "10.1007/978-3-319-46976-8_22",

language = "English",

isbn = "978-3-319-46975-1",

series = "Lecture notes in computer science",

publisher = "Springer",

pages = "209--218",

editor = "Gustavo Carneiro and Diana Mateus and Lo{\"i}c Peter and Andrew Bradley and Tavares, {Jo{\~a}o Manuel R. S.} and Vasileios Belagiannis and Papa, {Jo{\~a}o Paulo} and Nascimento, {Jacinto C.} and Marco Loog and Zhi Lu and Cardoso, {Jaime S.} and Juilen Cornebise",

booktitle = "Deep Learning and Data Labeling for Medical Applications",

note = "1st International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis ; Conference date: 21-10-2016 Through 21-10-2016",

}

TY - GEN

T1 - Early experiences with crowdsourcing airway annotations in chest CT

AU - Cheplygina, Veronika

AU - Perez-Rovira, Adria

AU - Kuo, Wieying

AU - Tiddens, Harm A. W. M.

AU - de Bruijne, Marleen

N1 - Conference code: 1

PY - 2016

Y1 - 2016

N2 - Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.

AB - Measuring airways in chest computed tomography (CT) images is important for characterizing diseases such as cystic fibrosis, yet very time-consuming to perform manually. Machine learning algorithms offer an alternative, but need large sets of annotated data to perform well. We investigate whether crowdsourcing can be used to gather airway annotations which can serve directly for measuring the airways, or as training data for the algorithms. We generate image slices at known locations of airways and request untrained crowd workers to outline the airway lumen and airway wall. Our results show that the workers are able to interpret the images, but that the instructions are too complex, leading to many unusable annotations. After excluding unusable annotations, quantitative results show medium to high correlations with expert measurements of the airways. Based on this positive experience, we describe a number of further research directions and provide insight into the challenges of crowdsourcing in medical images from the perspective of first-time users.

U2 - 10.1007/978-3-319-46976-8_22

DO - 10.1007/978-3-319-46976-8_22

M3 - Article in proceedings

AN - SCOPUS:84992489314

SN - 978-3-319-46975-1

T3 - Lecture notes in computer science

SP - 209

EP - 218

BT - Deep Learning and Data Labeling for Medical Applications

A2 - Carneiro, Gustavo

A2 - Mateus, Diana

A2 - Peter, Loïc

A2 - Bradley, Andrew

A2 - Tavares, João Manuel R. S.

A2 - Belagiannis, Vasileios

A2 - Papa, João Paulo

A2 - Nascimento, Jacinto C.

A2 - Loog, Marco

A2 - Lu, Zhi

A2 - Cardoso, Jaime S.

A2 - Cornebise, Juilen

PB - Springer

T2 - 1st International Workshop on Large-Scale Annotation of Biomedical Data and Expert Label Synthesis

Y2 - 21 October 2016 through 21 October 2016

ER -

Early experiences with crowdsourcing airway annotations in chest CT

Abstract

Konference

Adgang til dokumentet

Fingeraftryk

Citationsformater