Motion perception-driven multimodal self-supervised video object segmentation

Unsupervised video object segmentation segments foreground objects from videos without annotations. However, existing methods rely mainly on a single modality to process motion information and perform poorly with occlusions and static objects. To address this problem, we propose a multimodal motion perception network (M2PNet) based on self-supervised training for completely unlabeled video object segmentation tasks. M2PNet adopts a unique dual-path encoder–decoder structure to model spatial and temporal features to capture richer spatiotemporal information. Specifically, the spatial path computes spatial contrast matrices based on coattention mechanisms to enhance motion-related region correlations and accurately represent motion areas. The temporal path first utilizes residual connections and attention mechanisms to strengthen fused representations of different modalities and then iteratively learns motion change patterns based on slot attention to capture motion characteristics. In addition, we innovatively use residual maps to capture subtle interframe changes and build connections between different modal features. Extensive experiments on public datasets demonstrate that our multimodal method achieves significant improvements over other methods, demonstrating the advantage of multisource feature deep fusion. Our implementation is available at https://github.com/cao3082423114/M2PNet.