Scholarlink

<p>Multimodal combines multiple modalities to create insightful conclusions or to make more precise predictions. Nowadays, the multimodal concept is used to identify human emotions precisely. This study proposes a three-stage novel M-fusHER (Multimodal fusion Human Emotion Recognition) multimodal model for human emotion recognition in real-time with the help of text, audio, and videos. In the first stage, features are extracted with the help of a convolutional neural network merged with multiplicative LSTM. In the second stage, video and audio data, text, and audio are fused in binary form. In the third stage, real-time object detection for human emotion recognition on real videos is implemented. The experimental results are obtained by fusing audio, text, and videos by considering the standard features. For object detection, a fine-tuned YOLOv6 model was used for detecting facial features and expressions from the video. The multiplicative LSTM is also used to extract and learn from the text features. Three datasets, i.e., IEMOCAP, MOSEI, and MELD are used for implementation, and the detection accuracy of the proposed model M-<i>fus</i>HER on IEMOCAP, MOSEI, and MELD datasets is 95.45%, 88.76%, and 95.41% approximately, which is quite encouraging.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A multimodal fusion model for real-time environment emotion recognition using audio-visual-textual features

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年11月18日

查看全文
  • Chhaya Gupta,
  • Nasib Singh Gill,
  • Preeti Gulia,
  • Abhinav Kumar,
  • Hanen Karamti,
  • Demmelash Mollalign Moges,
  • Imen Safra

摘要

Multimodal combines multiple modalities to create insightful conclusions or to make more precise predictions. Nowadays, the multimodal concept is used to identify human emotions precisely. This study proposes a three-stage novel M-fusHER (Multimodal fusion Human Emotion Recognition) multimodal model for human emotion recognition in real-time with the help of text, audio, and videos. In the first stage, features are extracted with the help of a convolutional neural network merged with multiplicative LSTM. In the second stage, video and audio data, text, and audio are fused in binary form. In the third stage, real-time object detection for human emotion recognition on real videos is implemented. The experimental results are obtained by fusing audio, text, and videos by considering the standard features. For object detection, a fine-tuned YOLOv6 model was used for detecting facial features and expressions from the video. The multiplicative LSTM is also used to extract and learn from the text features. Three datasets, i.e., IEMOCAP, MOSEI, and MELD are used for implementation, and the detection accuracy of the proposed model M-fusHER on IEMOCAP, MOSEI, and MELD datasets is 95.45%, 88.76%, and 95.41% approximately, which is quite encouraging.