Scholarlink

<p>Speech emotion recognition has seen a surge in transformer models, which excel at understanding the overall message by analyzing long-term patterns in speech. However, these models come at a computational cost. In contrast, convolutional neural networks are faster but struggle with capturing these long-range relationships. Our proposed system, <i>MemoCMT</i>, tackles this challenge using a novel “cross-modal transformer” (<i>CMT</i>). This <i>CMT</i> can effectively analyze local and global speech features and their corresponding text. To boost efficiency, <i>MemoCMT</i> leverages recent advancements in pre-trained models: <i>HuBERT</i> extracts meaningful features from the audio, while <i>BERT</i> analyzes the text. The core innovation lies in how the <i>CMT</i> component utilizes and integrates these audio and text features. After this integration, different fusion techniques are applied before final emotion classification. Experiments show that <i>MemoCMT</i> achieves impressive performance, with the <i>CMT</i> using min aggregation achieving the highest unweighted accuracy (<i>UW-Acc</i>) of 81.33% and 91.93%, and weighted accuracy (<i>W-Acc</i>) of 81.85% and 91.84% respectively on benchmark IEMOCAP and ESD corpora. The results of our system demonstrate the generalization capacity and robustness for real-world industrial applications. Moreover, the implementation details of <i>MemoCMT</i> are publicly available at <a href="https://github.com/tpnam0901/MemoCMT/">https://github.com/tpnam0901/MemoCMT/</a> for reproducibility purposes.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

MemoCMT: multimodal emotion recognition using cross-modal transformer-based feature fusion

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年2月14日

查看全文
  • Mustaqeem Khan,
  • Phuong-Nam Tran,
  • Nhat Truong Pham,
  • Abdulmotaleb El Saddik,
  • Alice Othmani

摘要

Speech emotion recognition has seen a surge in transformer models, which excel at understanding the overall message by analyzing long-term patterns in speech. However, these models come at a computational cost. In contrast, convolutional neural networks are faster but struggle with capturing these long-range relationships. Our proposed system, MemoCMT, tackles this challenge using a novel “cross-modal transformer” (CMT). This CMT can effectively analyze local and global speech features and their corresponding text. To boost efficiency, MemoCMT leverages recent advancements in pre-trained models: HuBERT extracts meaningful features from the audio, while BERT analyzes the text. The core innovation lies in how the CMT component utilizes and integrates these audio and text features. After this integration, different fusion techniques are applied before final emotion classification. Experiments show that MemoCMT achieves impressive performance, with the CMT using min aggregation achieving the highest unweighted accuracy (UW-Acc) of 81.33% and 91.93%, and weighted accuracy (W-Acc) of 81.85% and 91.84% respectively on benchmark IEMOCAP and ESD corpora. The results of our system demonstrate the generalization capacity and robustness for real-world industrial applications. Moreover, the implementation details of MemoCMT are publicly available at https://github.com/tpnam0901/MemoCMT/ for reproducibility purposes.