Temporal Coordination of Facial Expressions and Head Movements in First Encounter Dialogues

TY - GEN

T1 - Temporal Coordination of Facial Expressions and Head Movements in First Encounter Dialogues

AU - Paggio, Patrizia

AU - Navarretta, Costanza

PY - 2018

Y1 - 2018

N2 - This paper deals with the temporal coordination between facial expressions and co-occurring head movements in a multimodal corpus of first encounter conversations. In particular, we look at how the onset of facial expressions is coordinated with the first overlapping head movement, in other words which of the two modalities precedes the other and why. We find and discuss statistical main effects on the temporal delays between the two behaviours due to individual variation, type of head movement, and the communicative function of the multimodal signal. In particular, the analysis shows that when speakers give feedback, their facial expression becomes visible before the head starts to move, especially in the case of negative comments associated with frowning or scowling. The opposite is true when the multimodal signal is used as a comment to the speaker's own speech. The motivation for the analysis is to shed light on a less studied aspect of multimodal communication -- an aspect that is relevant to the generation of natural multimodal expressions in ECAs.

AB - This paper deals with the temporal coordination between facial expressions and co-occurring head movements in a multimodal corpus of first encounter conversations. In particular, we look at how the onset of facial expressions is coordinated with the first overlapping head movement, in other words which of the two modalities precedes the other and why. We find and discuss statistical main effects on the temporal delays between the two behaviours due to individual variation, type of head movement, and the communicative function of the multimodal signal. In particular, the analysis shows that when speakers give feedback, their facial expression becomes visible before the head starts to move, especially in the case of negative comments associated with frowning or scowling. The opposite is true when the multimodal signal is used as a comment to the speaker's own speech. The motivation for the analysis is to shed light on a less studied aspect of multimodal communication -- an aspect that is relevant to the generation of natural multimodal expressions in ECAs.

M3 - Article in proceedings

SP - 40

EP - 44

BT - Proceedings of the LREC 2018 Workshop "LB-ILR2018 and MMC2018 Joint Workshop"

A2 - Koiso, Hanae

A2 - Paggio, Patrizia

PB - European Language Resources Association

ER -