Dual-stream network with cross-layer attention and similarity constraint for micro-expression recognition

Micro-expression recognition (MER) is a pivotal research area within human emotion analysis. However, the fleeting, subtle, and complex nature of micro-expressions poses challenges for accurate and efficient recognition. To address this, this paper proposes a Dual-Stream Network with Cross-layer Attention and Similarity Constraint (DSN-CASC) for MER. The network is designed with two parallel branches, each dedicated to learning features of stacked optical flow maps and independent micro-expression optical flow components. The network amplifies its focus on representing features at different hierarchical levels via incorporating a cross-layer attention module, thereby enhancing the capture of key features associated with micro-expression variations. Furthermore, a similarity constraint strategy is introduced to ensure that the micro-expression features extracted by the dual branches exhibit similar representations, improving the network’s overall representation capability. Finally, a simple feature fusion approach is employed for micro-expression classification. Extensive experiments on the composite database validate the effectiveness of DSN-CASC under leave-one-subject-out cross-validation and composite database evaluation protocol. The results demonstrate that our proposed approach achieves promising performance improvements, which provide new insights and effective solutions for MER research.