Accurate and objective pain assessment is crucial for effective pain management. This paper proposes a novel multimodal deep learning framework for automatic pain detection using a hybrid architecture with feature-level fusion. The framework leverages multimodal data including facial expressions and physiological signals (EDA and ECG) from the BioVid Heat Pain database (Part A). The novel hybrid architecture consists of two streams as stream 1 employs an attention-based CNN-LSTM to extract features from facial expressions videos, capture temporal dependencies, and focus on relevant aspects of the video data, and stream 2 with an LSTM to capture temporal patterns in the physiological signals. The performance of the proposed model was examined in both unimodal and multimodal settings. In a binary classification task distinguishing No Pain from Severe Pain, electrodermal activity (EDA) outperformed all other single data sources, achieving high average accuracy (83.05% for 67 subjects and 82.69% for 87 subjects) and F1-scores (81.66 and 80.18, respectively) using k-fold cross-validation. Additionally, the multimodal setting (Video + EDA) achieved higher accuracy (84.15% for 67 subjects and 83.35% for 87 subjects) and F1-scores (82.86 and 82.36, respectively).

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal Automatic Acute Pain Recognition Using Facial Expressions and Physiological Signals

  • Jaleh Farmani,
  • Alessandro Giuseppi,
  • Ghazal Bargshady,
  • Raul Fernandez Rojas

摘要

Accurate and objective pain assessment is crucial for effective pain management. This paper proposes a novel multimodal deep learning framework for automatic pain detection using a hybrid architecture with feature-level fusion. The framework leverages multimodal data including facial expressions and physiological signals (EDA and ECG) from the BioVid Heat Pain database (Part A). The novel hybrid architecture consists of two streams as stream 1 employs an attention-based CNN-LSTM to extract features from facial expressions videos, capture temporal dependencies, and focus on relevant aspects of the video data, and stream 2 with an LSTM to capture temporal patterns in the physiological signals. The performance of the proposed model was examined in both unimodal and multimodal settings. In a binary classification task distinguishing No Pain from Severe Pain, electrodermal activity (EDA) outperformed all other single data sources, achieving high average accuracy (83.05% for 67 subjects and 82.69% for 87 subjects) and F1-scores (81.66 and 80.18, respectively) using k-fold cross-validation. Additionally, the multimodal setting (Video + EDA) achieved higher accuracy (84.15% for 67 subjects and 83.35% for 87 subjects) and F1-scores (82.86 and 82.36, respectively).