Audio–visual segmentation via hierarchical side tuning with state space model
摘要
Audio-Visual Segmentation (AVS) is a task aimed at predicting pixel-level masks for sound-producing objects in videos. Existing AVS methods typically involve a heavy audio encoder and use full fine-tuning or additional structures like adapters on the visual branch. We argue that these approaches incur significant training costs and may disrupt the pretrained model’s prior knowledge. To address this, we propose a novel hierarchical side-tuning framework for AVS, utilizing a side network that simultaneously performs audio encoding and cross-domain adaptation. By freezing the visual encoder and only tuning the side network, we significantly reduce the number of parameters to be trained. Additionally, inspired by the recent success of state space models (SSMs), we introduce an Audio-Visual Fusion (AVF) module in the side network and design a lightweight SSM-based decoder to enhance feature fusion and decoding. Experimental results on multiple AVS datasets demonstrate that our method achieves competitive or even superior performance compared with state-of-the-art approaches, while using fewer learnable parameters.