Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals objectively reflect the spatiotemporal dynamics of brain activity during emotional cognition. Combining both modalities can significantly improve emotion-recognition accuracy. Due to its powerful capability in representing graph structures, the Graph Isomorphism Network (GIN) can effectively capture relationships between nodes and topological features. In this study, we present TAS-GIN, a Two-Hop Attention Symmetric GIN for multimodal emotion recognition. We first construct separate EEG and fNIRS graphs and introduce a two-hop neighbor aggregation strategy to expand each node’s receptive field. A graph attention mechanism is incorporated to adaptively assign weights to different neighbors, highlighting the contribution of key nodes. Additionally, a symmetric mapping module captures complementary features between the left and right hemispheres. Experimental results demonstrate that TAS-GIN achieves higher accuracy in multimodal emotion recognition using EEG and fNIRS, with improvements of 3.76% and 16.27% over EEG-only and fNIRS-only modalities, respectively. Ablation studies further confirm that TAS-GIN outperforms other commonly used GIN aggregation variants, yielding an average accuracy improvement of 1.4% to 2.6%.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

EEG and fNIRS-Based Emotion Recognition Using an Improved Graph Isomorphism Network

  • Bingzhen Yu,
  • Xueying Zhang,
  • Guijun Chen,
  • Fenglian Li

摘要

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals objectively reflect the spatiotemporal dynamics of brain activity during emotional cognition. Combining both modalities can significantly improve emotion-recognition accuracy. Due to its powerful capability in representing graph structures, the Graph Isomorphism Network (GIN) can effectively capture relationships between nodes and topological features. In this study, we present TAS-GIN, a Two-Hop Attention Symmetric GIN for multimodal emotion recognition. We first construct separate EEG and fNIRS graphs and introduce a two-hop neighbor aggregation strategy to expand each node’s receptive field. A graph attention mechanism is incorporated to adaptively assign weights to different neighbors, highlighting the contribution of key nodes. Additionally, a symmetric mapping module captures complementary features between the left and right hemispheres. Experimental results demonstrate that TAS-GIN achieves higher accuracy in multimodal emotion recognition using EEG and fNIRS, with improvements of 3.76% and 16.27% over EEG-only and fNIRS-only modalities, respectively. Ablation studies further confirm that TAS-GIN outperforms other commonly used GIN aggregation variants, yielding an average accuracy improvement of 1.4% to 2.6%.