Micro-expression (ME), as a specific spontaneous and unconscious facial expression, reveals human genuine thoughts and intentions. Due to the brief duration and subtle amplitude, it is extraordinarily challengeable to spot ME from an ordinary long video, especially confused with the Macro-expression (MaE). To address this issue, this paper proposes a Multi-level Cross-attention Swin Transformer Fusion model for diverse detail feature representation on ME spotting, namely MCSTF. Specifically, this model first crops and aligns the input video frames, then yields optical flow patterns between two adjacent frames, concentrating on specific facial regions with the Region of Interest (ROI) operator. Subsequently, these preprocessed features are separately fed into the multi-stream convolutional neural network channel and the multi-level channel for diverse subtle features extraction. The cross-attention mechanism is embedded based on the Swin Transformer to facilitate local feature interaction for cross-channel relations. Moreover, the hierarchical maps from different stages are progressively aggregated to enhance the global representation on ME. In the end, the output component of the MCSTF model is to predict the probability of a particular frame corresponding to a specific expression. Extensive comparison experiments show that the proposed MCSTF model achieves the efficient local and global feature fusion of hierarchical optical flow features, which outperforms other ME spotting methods based on deep neural networks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Macro and Micro-Expressions Spotting Based on Cross-Attention Transformer Progressive Fusion Network

  • Qinqin Lv,
  • Zhiwu Zhou,
  • ChenHao Li,
  • Zhihua Xie

摘要

Micro-expression (ME), as a specific spontaneous and unconscious facial expression, reveals human genuine thoughts and intentions. Due to the brief duration and subtle amplitude, it is extraordinarily challengeable to spot ME from an ordinary long video, especially confused with the Macro-expression (MaE). To address this issue, this paper proposes a Multi-level Cross-attention Swin Transformer Fusion model for diverse detail feature representation on ME spotting, namely MCSTF. Specifically, this model first crops and aligns the input video frames, then yields optical flow patterns between two adjacent frames, concentrating on specific facial regions with the Region of Interest (ROI) operator. Subsequently, these preprocessed features are separately fed into the multi-stream convolutional neural network channel and the multi-level channel for diverse subtle features extraction. The cross-attention mechanism is embedded based on the Swin Transformer to facilitate local feature interaction for cross-channel relations. Moreover, the hierarchical maps from different stages are progressively aggregated to enhance the global representation on ME. In the end, the output component of the MCSTF model is to predict the probability of a particular frame corresponding to a specific expression. Extensive comparison experiments show that the proposed MCSTF model achieves the efficient local and global feature fusion of hierarchical optical flow features, which outperforms other ME spotting methods based on deep neural networks.