An Agentic Framework for 3D Semantic Gaussian Maps

Egocentric 3D object localization driven by natural language is a pivotal capability for next-generation Embodied AI and Augmented Reality (AR) assistants. However, existing approaches often treat linguistic reasoning, 2D visual grounding, and 3D scene reconstruction as isolated tasks, hindering the ability to intuitively locate objects in physical space through speech alone. To bridge this gap, we present a novel unified agentic framework that seamlessly integrates Large Language Model (LLM) agents with video understanding and semantic 3D reconstruction. Our system captures egocentric video and user speech, transcribing audio commands into text via Automatic Speech Recognition (ASR). We employ LangManus as the central intelligent agent to interpret user intent and orchestrate task planning. Upon receiving a localization query, the system utilizes Sa2VA (Marrying SAM2 with LLaVA) to perform dense grounded understanding within the video stream, while concurrently deploying SegAnyGaussian (SAGA) to reconstruct the scene and lift 2D segmentation features into a semantically segmented 3D Gaussian Splatting field. By synergizing the spatiotemporal reasoning of Sa2VA with the interactive 3D semantic capabilities of SAGA, our framework effectively translates spoken instructions into precise 3D object coordinates. This end-to-end pipeline significantly enhances human-environment interaction, enabling accurate and intuitive object retrieval in complex 3D scenarios.