Predicting Speech Overlaps from Speech Tokens and Co-occurring Body Behaviours in Dyadic Conversations

TY - GEN

T1 - Predicting Speech Overlaps from Speech Tokens and Co-occurring Body Behaviours in Dyadic Conversations

AU - Navarretta, Costanza

PY - 2013

Y1 - 2013

N2 - This paper deals with speech overlaps in dyadic video record-ed spontaneous conversations. Speech overlaps are quite common in everyday conversations and it is therefore important to study their occurrences in different communicative situations and settings and to model them in applied communicative systems. In the present work, we wanted to investigate the frequency and use of speech overlaps in a multimodally annotated corpus of first encounters. Speech overlaps were automatically tagged and a Bayesian Network learner was trained on the multimodal annotations in order to determine to which extent overlaps can be predicted so they can be dealt with in conversational devices and to investigate the relation between overlaps, speech tokens and co-occurring body behaviours. The annotations comprise shape and functions of head movements, facial expressions and body postures. 23% of the speech tokens and 90% of the spoken contributions of the first encounters are overlapping. The best classification results were obtained training the classifier on multimodal behaviours (speech and co-occurring head movements, facial expressions and body postures) which surround-ed the overlaps. Training the classifier on all speech tokens also gave good results while adding the shape of co-occurring body behaviours to them did not affect the results. Thus, the behaviours of the conversation participants does not change when there is a speech overlap. This could indicate that most of the overlaps in the first encounters are non competitive.

AB - This paper deals with speech overlaps in dyadic video record-ed spontaneous conversations. Speech overlaps are quite common in everyday conversations and it is therefore important to study their occurrences in different communicative situations and settings and to model them in applied communicative systems. In the present work, we wanted to investigate the frequency and use of speech overlaps in a multimodally annotated corpus of first encounters. Speech overlaps were automatically tagged and a Bayesian Network learner was trained on the multimodal annotations in order to determine to which extent overlaps can be predicted so they can be dealt with in conversational devices and to investigate the relation between overlaps, speech tokens and co-occurring body behaviours. The annotations comprise shape and functions of head movements, facial expressions and body postures. 23% of the speech tokens and 90% of the spoken contributions of the first encounters are overlapping. The best classification results were obtained training the classifier on multimodal behaviours (speech and co-occurring head movements, facial expressions and body postures) which surround-ed the overlaps. Training the classifier on all speech tokens also gave good results while adding the shape of co-occurring body behaviours to them did not affect the results. Thus, the behaviours of the conversation participants does not change when there is a speech overlap. This could indicate that most of the overlaps in the first encounters are non competitive.

M3 - Article in proceedings

SP - 157

EP - 163

BT - Proceedings of the 15th ACM International Conference on Multimodal Interaction (ICMI 2013)

PB - Association for Computing Machinery

ER -