Scholarlink

Human emotions entail a complex set of behavioral, physiological and cognitive changes. Current state-of-the-art models fuse the behavioral and physiological components using classic machine learning, rather than recent deep learning techniques. We propose to fill this gap, designing the Multimodal for Video and Physio (MVP) architecture, streamlined to fuse video and physiological signals. Differently then others approaches, MVP exploits the benefits of attention to enable the use of long input sequences (1–2 min). We have studied video and physiological backbones for inputting long sequences and evaluated our method with respect to the state-of-the-art. Our results show that MVP outperforms former methods for emotion recognition based on facial videos, EDA, and ECG/PPG. The code is available on GitHub ( https://github.com/EmotionLab/EmotionVMAE ).

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

MVP: Multimodal Emotion Recognition Based on Video and Physiological Signals

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Valeriya Strizhkova,
  • Hadi Kachmar,
  • Hava Chaptoukaev,
  • Raphael Kalandadze,
  • Natia Kukhilava,
  • Tatia Tsmindashvili,
  • Nibras Abo-Alzahab,
  • Maria A. Zuluaga,
  • Michal Balazia,
  • Antitza Dantcheva,
  • François Brémond,
  • Laura M. Ferrari

摘要

Human emotions entail a complex set of behavioral, physiological and cognitive changes. Current state-of-the-art models fuse the behavioral and physiological components using classic machine learning, rather than recent deep learning techniques. We propose to fill this gap, designing the Multimodal for Video and Physio (MVP) architecture, streamlined to fuse video and physiological signals. Differently then others approaches, MVP exploits the benefits of attention to enable the use of long input sequences (1–2 min). We have studied video and physiological backbones for inputting long sequences and evaluated our method with respect to the state-of-the-art. Our results show that MVP outperforms former methods for emotion recognition based on facial videos, EDA, and ECG/PPG. The code is available on GitHub ( https://github.com/EmotionLab/EmotionVMAE ).