Previous Audio-Visual Source Separation methods often rely on simple concatenation or element-wise multiplication for multi-modal fusion, limiting their ability to capture dynamic inter-modal relationships. With the inclusion of emerging modalities like motion, traditional single-stage fusion strategies face clear challenges. To overcome this, we propose a streaming multi-modal dynamic fusion approach based on a gating mechanism, which progressively integrates visual, auditory, and motion features. A visual attention module is also introduced to enhance visual representations. Experiments on the MUSIC and MUSIC21 datasets show that our method significantly outperforms existing approaches, demonstrating the effectiveness of the proposed gating and multi-stage fusion strategies.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

SMFNet: Streaming Multi-modal Audio-Visual Source Separation via Gated Fusion

  • Shiyu Sun,
  • Meiling Zhu,
  • Yinfeng Yu

摘要

Previous Audio-Visual Source Separation methods often rely on simple concatenation or element-wise multiplication for multi-modal fusion, limiting their ability to capture dynamic inter-modal relationships. With the inclusion of emerging modalities like motion, traditional single-stage fusion strategies face clear challenges. To overcome this, we propose a streaming multi-modal dynamic fusion approach based on a gating mechanism, which progressively integrates visual, auditory, and motion features. A visual attention module is also introduced to enhance visual representations. Experiments on the MUSIC and MUSIC21 datasets show that our method significantly outperforms existing approaches, demonstrating the effectiveness of the proposed gating and multi-stage fusion strategies.