Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) are non-invasive neuro-imaging techniques that can be acquired synchronously. EEG captures electrophysiological activity, while fNIRS measures hemodynamic responses, providing complementary insights into neural processes. The integration of both modalities enhances emotion recognition, yet most existing methods rely on static electrode arrangements, which restrict the adaptability of functional brain networks across varying emotional states and limit effective bimodal fusion. To address these limitations, this paper proposes an Electrode Dynamic Permutation and Hierarchical Fusion (EDPHF) model for bimodal emotion recognition. EEG and fNIRS signals are modeled as graphs, with an end-to-end learnable dynamic adjacency matrix enabling adaptive electrode permutation. The model follows a three-stage cascade: extracting local spatial patterns via graph convolution and channel attention, capturing global dependencies with a Transformer, and adaptively fusing bimodal representations through modality attention. Experiments on a self-constructed dataset show that EDPHF outperforms state-of-the-art methods in recognition accuracy, validating its dynamic electrode permutation and hierarchical fusion strategies.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Joint Dynamic Electrode Permutation and Hierarchical Fusion for EEG-fNIRS Emotion Classification

  • Yaru Zhou,
  • Xuan Meng,
  • Xueying Zhang

摘要

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) are non-invasive neuro-imaging techniques that can be acquired synchronously. EEG captures electrophysiological activity, while fNIRS measures hemodynamic responses, providing complementary insights into neural processes. The integration of both modalities enhances emotion recognition, yet most existing methods rely on static electrode arrangements, which restrict the adaptability of functional brain networks across varying emotional states and limit effective bimodal fusion. To address these limitations, this paper proposes an Electrode Dynamic Permutation and Hierarchical Fusion (EDPHF) model for bimodal emotion recognition. EEG and fNIRS signals are modeled as graphs, with an end-to-end learnable dynamic adjacency matrix enabling adaptive electrode permutation. The model follows a three-stage cascade: extracting local spatial patterns via graph convolution and channel attention, capturing global dependencies with a Transformer, and adaptively fusing bimodal representations through modality attention. Experiments on a self-constructed dataset show that EDPHF outperforms state-of-the-art methods in recognition accuracy, validating its dynamic electrode permutation and hierarchical fusion strategies.