Recently, emotion recognition technology has gained widespread attention in education. This technology plays a crucial role in enhancing learning motivation, engagement, teacher-student relationships, and academic achievement. With the rapid development of deep learning, multimodal emotion recognition (MER) systems have emerged, capable of effectively integrating information from various data sources, such as audio, video, and text, thus providing more accurate emotion analysis. We propose an innovative hybrid modality emotion recognition model that combines facial expressions and blood volume pulse (BVP) physiological signals, employing a decision fusion strategy to generate the final emotion recognition results. Experimental validation on a self-constructed hybrid modality database revealed that the recognition accuracy of the model under single modalities (facial expressions or physiological signals) was 64.32% and 79.33%, respectively. However, when the two modalities were fused, the emotion recognition accuracy significantly improved to 84.55%. This result demonstrates the effectiveness of our model in multimodal emotion recognition tasks. Our research highlights the tremendous potential of MER technology and provides a more precise emotion analysis tool for the education, aiding in the creation of more intelligent and personalized learning environments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Emotion Recognition for Learners Based on Facial Expressions and BVP Signals

  • Kai Yang,
  • Yantao Wei,
  • Qi Xu,
  • Tao Hu,
  • Peishu Chang

摘要

Recently, emotion recognition technology has gained widespread attention in education. This technology plays a crucial role in enhancing learning motivation, engagement, teacher-student relationships, and academic achievement. With the rapid development of deep learning, multimodal emotion recognition (MER) systems have emerged, capable of effectively integrating information from various data sources, such as audio, video, and text, thus providing more accurate emotion analysis. We propose an innovative hybrid modality emotion recognition model that combines facial expressions and blood volume pulse (BVP) physiological signals, employing a decision fusion strategy to generate the final emotion recognition results. Experimental validation on a self-constructed hybrid modality database revealed that the recognition accuracy of the model under single modalities (facial expressions or physiological signals) was 64.32% and 79.33%, respectively. However, when the two modalities were fused, the emotion recognition accuracy significantly improved to 84.55%. This result demonstrates the effectiveness of our model in multimodal emotion recognition tasks. Our research highlights the tremendous potential of MER technology and provides a more precise emotion analysis tool for the education, aiding in the creation of more intelligent and personalized learning environments.