Harmony AI of Senses: Therapeutic Companion through Multimodal Integration
摘要
The purpose of this study is to develop a sophisticated environment that combines capabilities of Large Language Models (LLMs), Sentiment analysis, and computer vision. The goal of this proposed environment is to create a novel conversational agent capable of understanding and generating interactive dialogues while considering emotions through textual, verbal, and visual inputs. Our model is built through Fine-tuning LLMs on an actual and synthetic dataset of therapeutic conversation. The project follows a multi-faceted approach, involving the integration of several novel and state-of-the-art models for multi-modality: The “Harmony AI” has great implications for the alarming high rates of depression and loneliness all over the globe along with far-reaching implications for various applications, including minor therapeutic, customer service, and educational tools. Its ability to comprehend textual, verbal, and visual cues facilitates more natural and engaging interactions and experiences, thus potentially supporting and improving the mental health of individuals. Our multimodal architecture comprises of: The multimodal and emotion-based graphs collectively contribute to the system’s versatility and effectiveness. The study substantiates the success of this multimodal approach with user study ratings, revealing commendable scores in Helpfulness (3.50/5), Emotionally Safe (4.01/5), and Recommendation (3.38/5). This research not only pushes the boundaries of AI-driven conversational agents but also sheds light on the potential of multimodal integration for creating a versatile and user-friendly virtual therapeutic environment.