More Than One Gesture but Less Than Two? Inter-stroke Dependencies in Form and Meaning

In gesture analysis, one must segment the gesture stream into discrete units to be analyzed for meaning and alignment with accompanying speech. This requires defining what constitutes a single “gesture”. Following the influential work of Kendon (2004) and McNeill (2005), this nearly always involves identifying the “stroke”, the gesture’s meaningful core. Innovations in the understanding of gestural meaning complicate this procedure, as there is increasing evidence that gestures are both internally complex and compositional. In the present work, we discuss the ways in which this complexity in gestural meaning may be addressed by including the notion of “gesture sequences”. Using data from American television talk shows and TED talks, we demonstrate two types of inter-stroke dependencies in pragmatic gestures where one movement’s meaning and form is dependent on that of a previous movement: (i) “closing gestures” signal the closure of a discourse topic and are dependent on a preceding presentation gesture; (ii) “contrast sequences” signal a contrast discourse relation and are dependent on adjacent presentation gestures occurring in different regions of gesture space. These findings have important implications for both how we annotate the gesture stream and how we simulate gesture in virtual agents.