Scholarlink

<p>The proposed study presents an <i>EmotiFusion-7</i> multimodal framework for emotion recognition using a self-created and compiled comic panel Custom-10K dataset comprised of 10,000 images. The proposed framework includes an attention-based fusion of vision transformers (ViTs) and bidirectional encoder representations from transformers (BERT) for extracting visual features and capturing textual semantics, respectively. Employing a Custom-10K dataset of annotated comic panels along with the fusion mechanism, the proposed work achieved a state-of-the-art performance with an exceptional overall accuracy of 92.3% and an area under the curve-receiver operating characteristic (AUC-ROC) score of 0.96 over seven emotions classes. In addition to this, the proposed study demonstrates superior performance in comparison with existing methods, conducting robustness testing and cross-domain evaluation. The proposed research establishes a new standard for multimodal emotion recognition along with a future scope for broader applications in visual storytelling and document image analysis.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

EmotiFusion-7: a multimodal framework for emotion recognition in comics using vision transformers and BERT

  • 期刊论文

  • 发表日期：

    2025年3月24日

查看全文
  • Rishabh Sharma,
  • Vinay Kukreja

摘要

The proposed study presents an EmotiFusion-7 multimodal framework for emotion recognition using a self-created and compiled comic panel Custom-10K dataset comprised of 10,000 images. The proposed framework includes an attention-based fusion of vision transformers (ViTs) and bidirectional encoder representations from transformers (BERT) for extracting visual features and capturing textual semantics, respectively. Employing a Custom-10K dataset of annotated comic panels along with the fusion mechanism, the proposed work achieved a state-of-the-art performance with an exceptional overall accuracy of 92.3% and an area under the curve-receiver operating characteristic (AUC-ROC) score of 0.96 over seven emotions classes. In addition to this, the proposed study demonstrates superior performance in comparison with existing methods, conducting robustness testing and cross-domain evaluation. The proposed research establishes a new standard for multimodal emotion recognition along with a future scope for broader applications in visual storytelling and document image analysis.