Scholarlink

<p>Early detection of emotional distress is critical for mental health, yet existing systems are often limited to two modalities, lack real-time web deployment, and rely on heuristic fusion weights. This work proposes a real-time, web-deployed multimodal emotion detection framework integrating three modalities: facial expressions via a 2D-CNN (98.48% validation accuracy on CK+), speech via a 1D-CNN (90.18% on RAVDESS+TESS+SAVEE), and text via a fine-tuned BERT transformer (84.77% on GoEmotions). The models are combined through an ablation-validated weighted decision-level late fusion strategy (Face: 0.35, Speech: 0.25, Text: 0.40), outperforming unimodal baselines and equal-weight fusion. A DeiT-based Vision Transformer was evaluated for FER but not adopted due to lower accuracy (73.96% on RAF-DB) and higher computational cost. The framework is deployed as the WellNest web platform for real-time monitoring and mental health journaling. Limitations, failure cases, and clinical validation needs are discussed.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A web-enabled real-time multimodal emotion detection system integrating BERT text embeddings and CNN-based speech and facial models

  • 期刊论文

  • 发表日期：

    2026年7月14日

查看全文
  • Sreeja Kamera,
  • Dhanusha Challa,
  • Hannah Bwalya,
  • D. Ramesh Reddy,
  • Prakash Kodali

摘要

Early detection of emotional distress is critical for mental health, yet existing systems are often limited to two modalities, lack real-time web deployment, and rely on heuristic fusion weights. This work proposes a real-time, web-deployed multimodal emotion detection framework integrating three modalities: facial expressions via a 2D-CNN (98.48% validation accuracy on CK+), speech via a 1D-CNN (90.18% on RAVDESS+TESS+SAVEE), and text via a fine-tuned BERT transformer (84.77% on GoEmotions). The models are combined through an ablation-validated weighted decision-level late fusion strategy (Face: 0.35, Speech: 0.25, Text: 0.40), outperforming unimodal baselines and equal-weight fusion. A DeiT-based Vision Transformer was evaluated for FER but not adopted due to lower accuracy (73.96% on RAF-DB) and higher computational cost. The framework is deployed as the WellNest web platform for real-time monitoring and mental health journaling. Limitations, failure cases, and clinical validation needs are discussed.