Show & Tell: Visual and Verbal Cues for Controlling Digital Content

Despite the prevalence of settings where multiple users are co-located in a given area, working towards a common goal, the systems and interfaces employed in these scenarios are often not specialized to the setting, and introduce inefficiencies and unneeded redundancies where tasks often need to be repeated across users. In this work we describe a set of interaction classes that are enabled by an ambient intelligent environment, where users can more naturally manipulate and control the digital content present in their operating domain (documents, media, data, viewpoints, etc.), through simultaneously showing the system (via gestures and visual cues) where to place content, and using natural language as a way of telling the system what content they mean. Using this system and our proposed interaction framework, we plan to conduct empirical user studies to further explore the possibilities offered by this multimodal interface, and how it can enable more effective and efficient task executions by both individuals and groups.