Exploring Audio-Visual Fusion for Sound Event Localization and Detection with BEATs
摘要
Human perception and environmental understanding fundamentally rely on visual and auditory as complementary sensory modalities. Consequently, Audio-Visual Sound Event Localization and Detection (AV-SELD) has emerged as an important research direction, aiming to overcome traditional SELD limitations through multimodal information fusion. However, existing approaches face challenges, including data scarcity and inadequate audio-visual fusion. Therefore, this paper proposes a novel AV-SELD-BEATs framework that effectively leverages knowledge from the pretrained model BEATs and introduces a fine-grained Audio-Visual fusion module (AVFusion). Specifically, to address the challenge of data scarcity, we introduce BEATs as an auxiliary audio feature extractor for the first time in AV-SELD, providing class-semantic-rich acoustic inputs for subsequent multimodal fusion. Building upon this foundation, we implement our previously developed Global Enhanced Frame Prompt Tuning (GE-FPT) method to further unleash the representational capabilities of BEATs. Furthermore, to effectively leverage audio-visual complementarity, we propose the AVFusion module, incorporating a fine-grained audio-visual non-local attention mechanism to capture fine-grained cross-modal correspondences. Extensive experiments conducted on the DCASE2025 Task 3 dataset prove that the AV-SELD-BEATs framework is effective. The experimental results show that the proposed framework achieves substantial performance gains over the DCASE2025 baseline system, with \(F1-score\) of 31.8 ( \(+\) 12.15) on the DCASE2025 Challenge Task 3 Dataset. These outcomes demonstrate the effectiveness of enhancing audio representations via BEATs and capturing cross-modal fusion correspondence through the AVFusion module.