Actions, Not Apps: Toward Using LLMs to Reshape Context Aware Interactions in Mixed Reality Systems

Mixed reality computing merges user perception of the environment with digital information. As we move from flatscreen computing toward head-mounted computing, the necessity for developing alternative interactions and user flows becomes more evident. Activity theory provides a holistic overview of user interactions and motives. In this work in progress, we propose Action Sandbox Workspace as an interaction framework for the future of MR systems by focusing on action-centric interactions rather than application-centric interactions, aiming to bridge the gap between user goals and system functionalities in everyday tasks. By integrating the ontology of actions, user intentions, and context and connecting it to spatial data mapping, this forward-looking framework aims to create a contextually adaptive user interaction environment. The recent development in large language models (LLMs) has made the implementation of this interaction flow feasible by enabling inference and decision-making based on text-based descriptions of a user’s state and intentions with data and actions users have access to. We propose this approach as a future direction for developing mixed reality platforms and integrating AI in interacting with computers.