TY - JOUR
T1 - Turn-taking in human face-to-face interaction is multimodal
T2 - gaze direction and manual gestures aid the coordination of turn transitions
AU - Kendrick, Kobin H
AU - Holler, Judith
AU - Levinson, Stephen C.
PY - 2023/4/24
Y1 - 2023/4/24
N2 - Human communicative interaction is characterized by rapid and precise turn-taking. This is achieved by an intricate system that has been elucidated in the field of conversation analysis, based largely on the study of the audi- tory signal. This model suggests that transitions occur at points of possible completion identified in terms of linguistic units. Despite this, considerable evidence exists that visible bodily actions including gaze and gestures also play a role. To reconcile disparate models and observations in the literature, we combine qualitative and quantitative methods to analyse turn-taking in a corpus of multimodal interaction using eye-trackers and multiple cameras. We show that transitions seem to be inhibited when a speaker averts their gaze at a point of possible turn completion, or when a speaker produces gestures which are beginning or unfinished at such points. We further show that while the direction of a speaker’s gaze does not affect the speed of transitions, the production of manual gestures does: turns with gestures have faster transitions. Our findings suggest that the coordination of transitions involves not only linguistic resources but also visual gestural ones and that the transition-relevance places in turns are multimodal in nature. This article is part of a discussion meeting issue ‘Face2face: advancing the science of social interaction’.
AB - Human communicative interaction is characterized by rapid and precise turn-taking. This is achieved by an intricate system that has been elucidated in the field of conversation analysis, based largely on the study of the audi- tory signal. This model suggests that transitions occur at points of possible completion identified in terms of linguistic units. Despite this, considerable evidence exists that visible bodily actions including gaze and gestures also play a role. To reconcile disparate models and observations in the literature, we combine qualitative and quantitative methods to analyse turn-taking in a corpus of multimodal interaction using eye-trackers and multiple cameras. We show that transitions seem to be inhibited when a speaker averts their gaze at a point of possible turn completion, or when a speaker produces gestures which are beginning or unfinished at such points. We further show that while the direction of a speaker’s gaze does not affect the speed of transitions, the production of manual gestures does: turns with gestures have faster transitions. Our findings suggest that the coordination of transitions involves not only linguistic resources but also visual gestural ones and that the transition-relevance places in turns are multimodal in nature. This article is part of a discussion meeting issue ‘Face2face: advancing the science of social interaction’.
U2 - 10.1098/rstb.2021.0473
DO - 10.1098/rstb.2021.0473
M3 - Article
SN - 1471-2970
VL - 378
JO - Philosophical Transactions of the Royal Society B: Biological Sciences
JF - Philosophical Transactions of the Royal Society B: Biological Sciences
IS - 1875
ER -