Music Emotion Recognition (MER) is a challenging task considering the nuances of defining emotions. While unimodal models provide a good baseline for MER, multimodal models are becoming fundamental to provide an in-depth description of emotions. Leveraging on the multimodal MERGE dataset, we investigate the power of audio-related deep embeddings, lyrics informed features, and music-aware cues in providing an informative set of features for low-impact computational learning models. Results confirm that multimodal fusion outperforms unimodal approaches. Moreover, different experiments highlight the positive contribution of genre metadata and the potential use of harmonic features for real-time computationally low-impact applications. These findings confirm the importance of multimodal integration for robust and interpretable emotion recognition systems, while opening up future directions, including advanced feature fusion, user-specific model adaptation (user-tuning), and multi-label emotion representation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Decoding Emotions: Multimodal Integration of Deep Embeddings, Lyrics and Music-Aware Cues

  • Alessia Novacco,
  • Francesca Gasparini,
  • Giulia Rizzi,
  • Aurora Saibene

摘要

Music Emotion Recognition (MER) is a challenging task considering the nuances of defining emotions. While unimodal models provide a good baseline for MER, multimodal models are becoming fundamental to provide an in-depth description of emotions. Leveraging on the multimodal MERGE dataset, we investigate the power of audio-related deep embeddings, lyrics informed features, and music-aware cues in providing an informative set of features for low-impact computational learning models. Results confirm that multimodal fusion outperforms unimodal approaches. Moreover, different experiments highlight the positive contribution of genre metadata and the potential use of harmonic features for real-time computationally low-impact applications. These findings confirm the importance of multimodal integration for robust and interpretable emotion recognition systems, while opening up future directions, including advanced feature fusion, user-specific model adaptation (user-tuning), and multi-label emotion representation.