Neuro-Symbolic LIDA’s Semantic Vision System
摘要
This paper explores the integration of a Vision-Language Model (VLM) and a cognitive architecture to create a context-aware, neuro-symbolic system capable of real-time media capture. The Learning Intelligent Decision Agent (LIDA), inspired by human cognition and the Global Workspace Theory, is paired with MobileCLIP, a state-of-the-art VLM that can interpret semantic information in visual data. This integration aims to enhance LIDA’s sensory system by providing robust, semantic understanding of real-world scenes while maintaining its decision-making and memory mechanisms. The system leverages MobileCLIP’s ability to process image-text embeddings for zero-shot activity recognition, and LIDA’s cognitive cycles enable adaptive behavior based on contextually relevant input. A proof-of-concept is demonstrated through a task where the system identifies and records specific actions during a track and field event. Several strategies for embedding comparison, including moving averages and cosine similarity, are evaluated for their effectiveness in accurately detecting the action. The results show that the integration of MobileCLIP with LIDA is promising, offering a scalable solution for real-time, context-aware decision-making. This work lays the foundation for future advancements in LIDA-based neuro-symbolic systems, particularly in environments requiring both perceptual understanding and adaptive cognitive functions. Future developments will focus on expanding the system’s capabilities to handle more complex tasks, integrating lifelong learning, and enhancing the robustness of decision-making processes.