Abstract
This paper describes a pilot study act to investigate the semiotic types of hand gestures in video-recorded speeches and their automatic classification. Gestures, which also comprise e.g. head movements and body posture, contribute to the successful delivery of the message by reinforcing what is expressed by speech or by adding new information to what is uttered. The automatic classification of the semiotic type of gestures from their shape description can contribute to their interpretation in human-human communication and in advanced multimodal interactive systems. We annotated and analysed hand gestures produced by Barack Obama during two speeches at the Annual White House Correspondent Dinners and found differences in the contexts in which various hand gesture types were used. Then, we trained machine learning algorithms to classify the semiotic type of the hand gestures. The F-score obtained by the best performing algorithm on the classification of four semiotic types is 0.59. Surprisingly, the shape feature that contributes mostly to classification is the trajectory of the left hand. The results of this study are promising, but they should be tested on more data of different type, produced by different speakers and in more languages.
| Original language | English |
|---|---|
| Title of host publication | Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018) |
| Number of pages | 6 |
| Place of Publication | Paris, France |
| Publisher | European Language Resources Association |
| Publication date | 2018 |
| Pages | 1067-72 |
| ISBN (Electronic) | 979-10-95546-00-9 |
| Publication status | Published - 2018 |