Visually-guided audio-visual aegmentation via multi-scale fusion and content-guided attention
摘要
Audio-Visual Segmentation aims to pixel-wise locate and segment sounding objects in videos driven by audio cues. However, current mainstream methods typically employ audio-centric Transformer frameworks that derive object queries primarily from audio features. However, the modality gap limits these audio-centric approaches from accurately and consistently identifying sounds in videos, as relying on temporal audio signals to resolve spatial segmentation tasks may lead to perceptual ambiguity and a loss of fine-grained visual details, particularly in complex acoustic environments. To address these challenges, this paper proposes a novel visually-guided framework incorporating a Multi-Scale Fusion module and a Content-Guided Attention Fusion mechanism to priorities visual information to generate visually-derived queries, which then interact with audio context within a Transformer decoder for deep semantic refinement. Extensive experiments on standard benchmarks demonstrate that our approach achieves outperforming performance.