An Integrated Analysis for Identifying Iconic Gestures in Human-Robot Interactions

Co-speech human gesture analysis is an important aspect for social conversational interactions involving human-robot interfaces. Co-speech gestures require synchronous integration of speech, human posture, and motions. Iconic gestures are a major subclass of co-speech gestures that express entities and actions by their attributes such as shape-contours, magnitude, and proximity using the synchronous motions of fingers, palms, and spoken phrases. The attributes of entities and actions correlate directly with the displayed contours. In this research, we describe an integrated technique that combines motion analysis to derive contours, synchronization of motion with speech to identify words corresponding to iconic gestures, and conceptual dependency of action words to drive iconic gestures. This technique models motion-sketched contour as a combination of synchronous color Petri net extended to model composite motions and contour-segment patterns. We present high-level algorithms and the corresponding implementation for the proposed technique and evaluate its performance. Performance results show approximately 90% recognition of simple contours, including closed contours.