A cross-modal audiovisual speech separation framework for SepFormer networks incorporating bidirectional attention mechanisms
摘要
Most current audiovisual separation models typically employ traditional feature concatenation methods for modality fusion. These models often rely on simple convolutional neural networks (CNNs) or recurrent neural networks (RNNs), which face challenges in optimizing long speech sequences and lack strong parallelization capabilities. This paper fully considers the correlation between visual and audio features, and introduces the CrossTransformer cross-modal fusion module based on multi-head attention. Combined with the SepFormer network framework, we propose a cross-modal audiovisual speech separation framework with bidirectional attention mechanism, named CrossTransformer-SepFormer Audiovisual Speech Separation (C-SAVSS). The proposed model utilizes self-attention and bidirectional attention mechanisms to effectively fuse and model features from both audio and visual modalities. Compared to traditional feature concatenation methods, this approach allows more efficient correlation of different modalities, thus improving the subsequent separation performance. The SepFormer separation network adopts a dual-path structure, addressing the efficiency issue of processing long sequences and enhancing parallelization capability. Experimental evaluations based on the Source-to-Distortion Ratio (SDR) metric were conducted on the VoxCeleb2 dataset. The results show that the proposed C-SAVSS model achieves an SDR value of 9.55 dB, which is 0.6 dB higher than the combination of multi-head attention mechanism with U-Net separation network. This demonstrates that the integration of the CrossTransformer module with the SepFormer separation network significantly improves the overall performance of the network compared to the multi-head attention and U-Net combination.