Scholarlink

<p>Affective computing aims at enabling intelligent systems to be able to comprehend and react to human feelings. The high level of lyrics, audio, and visual blends meant to convey messages particularly in music videos presents an effective medium of emotion recognition. It is expected that the proposed study will establish a new multimodal music emotion recognition framework that combines a Hierarchical Attention Network and Knowledge Distillation. Initially, an edited multimodal affective data set is created, which includes various types of modalities audio and video. Bandpass filter is placed on audio and video streams to eliminate the noise and normalize the data. In audio, Mel-Frequency Cepstral Coefficients (MFCCS) are utilized during feature extraction, whereas VGG16 is utilized in extracting features of video frames. Late fusion strategy then combines these extracted features to come up with a single representation. The given model is called Hierarchical Attention Mechanism-based Knowledge Distillation (HAM-KD), which is used to identify music emotion. Hierarchical attention mechanism to successfully encode emotionally salient features on both modalities, on major audio excerpts and expressive video frame. Knowledge Distillation is later used to relay the information to a light student model out of a high capacity, multimodal instructor model. This allows the student model to store the performance of the teacher and at the same time greatly decrease the computational complexity, which is applicable in both real-life and low-resource contexts. Python 3.10.1 was used to implement the proposed method. Thorough experiments show that the proposed solution of HAM-KD is superior over multimodal base architectures with better outcome in terms of accuracy, F1-score, recall, and precision of 95% or higher to 98% or higher. The results underscore the usefulness of integrating hierarchical attention and knowledge distillation in order to improve the strength and efficacy of multimodal emotion recognition systems. </p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multi modal music emotion recognition based on hierarchical attention network and knowledge distillation

  • 期刊论文

  • 发表日期：

    2026年1月5日

查看全文
  • Shuqi Li

摘要

Affective computing aims at enabling intelligent systems to be able to comprehend and react to human feelings. The high level of lyrics, audio, and visual blends meant to convey messages particularly in music videos presents an effective medium of emotion recognition. It is expected that the proposed study will establish a new multimodal music emotion recognition framework that combines a Hierarchical Attention Network and Knowledge Distillation. Initially, an edited multimodal affective data set is created, which includes various types of modalities audio and video. Bandpass filter is placed on audio and video streams to eliminate the noise and normalize the data. In audio, Mel-Frequency Cepstral Coefficients (MFCCS) are utilized during feature extraction, whereas VGG16 is utilized in extracting features of video frames. Late fusion strategy then combines these extracted features to come up with a single representation. The given model is called Hierarchical Attention Mechanism-based Knowledge Distillation (HAM-KD), which is used to identify music emotion. Hierarchical attention mechanism to successfully encode emotionally salient features on both modalities, on major audio excerpts and expressive video frame. Knowledge Distillation is later used to relay the information to a light student model out of a high capacity, multimodal instructor model. This allows the student model to store the performance of the teacher and at the same time greatly decrease the computational complexity, which is applicable in both real-life and low-resource contexts. Python 3.10.1 was used to implement the proposed method. Thorough experiments show that the proposed solution of HAM-KD is superior over multimodal base architectures with better outcome in terms of accuracy, F1-score, recall, and precision of 95% or higher to 98% or higher. The results underscore the usefulness of integrating hierarchical attention and knowledge distillation in order to improve the strength and efficacy of multimodal emotion recognition systems.