Multimodal Emotion Recognition (MER) has made significant progress with the advancements in Transformer-based models in effectively processing data from various sources in recent years. However, these models still face challenges like computational cost, capturing long-range relationships over multimodal data inputs, and multimodal output generation in personalized environments. To address this, we present SymphonyVision, a novel multi-modal system that leverages different models to understand and generate personalized, emotion-driven multimedia content, including music, images, and dialogues, aimed at supporting stress management and emotional regulation. The system integrates Large Language Models (LLMs) for text-based emotion recognition, Speech Emotion Recognition (SER) with a Bidirectional Long Short-Term Memory (Bi-LSTM) network for speech analysis, and two independent Diffusion Models for generating emotionally aligned images and music. Semantic alignment between user inputs and generated outputs is achieved through contrastive learning with dynamic loss adjustment, enhancing the relevance and quality of the content. Experimental results validate the system’s efficacy. For image generation, SymphonyVision achieves a Fréchet Inception Distance (FID) score of 15.432, with a recall of 0.970 and precision of 0.924. For music generation, it records a Fréchet Audio Distance (FAD) score of 18.765, with a MuLan similarity of 0.717 and a Cross-modal Retrieval Accuracy of 0.825. Additional metrics, including a Modality Gap of 0.085 and a Cross-modal Average Similarity Score of 0.782, highlight the robustness of our approach. To ensure practical utility, we developed an intuitive web interface, enabling high school users to seamlessly interact with the system.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

SymphonyVision: Diffusion-Based System for Multimodal Emotion Recognition and Generation

  • Claire Guo,
  • Ivy Zan

摘要

Multimodal Emotion Recognition (MER) has made significant progress with the advancements in Transformer-based models in effectively processing data from various sources in recent years. However, these models still face challenges like computational cost, capturing long-range relationships over multimodal data inputs, and multimodal output generation in personalized environments. To address this, we present SymphonyVision, a novel multi-modal system that leverages different models to understand and generate personalized, emotion-driven multimedia content, including music, images, and dialogues, aimed at supporting stress management and emotional regulation. The system integrates Large Language Models (LLMs) for text-based emotion recognition, Speech Emotion Recognition (SER) with a Bidirectional Long Short-Term Memory (Bi-LSTM) network for speech analysis, and two independent Diffusion Models for generating emotionally aligned images and music. Semantic alignment between user inputs and generated outputs is achieved through contrastive learning with dynamic loss adjustment, enhancing the relevance and quality of the content. Experimental results validate the system’s efficacy. For image generation, SymphonyVision achieves a Fréchet Inception Distance (FID) score of 15.432, with a recall of 0.970 and precision of 0.924. For music generation, it records a Fréchet Audio Distance (FAD) score of 18.765, with a MuLan similarity of 0.717 and a Cross-modal Retrieval Accuracy of 0.825. Additional metrics, including a Modality Gap of 0.085 and a Cross-modal Average Similarity Score of 0.782, highlight the robustness of our approach. To ensure practical utility, we developed an intuitive web interface, enabling high school users to seamlessly interact with the system.