Language-guided video object segmentation (LVOS) has achieved remarkable progress through advanced video segmentation models, but their performance heavily relies on large-scale fine-grained training data. The acquisition and fine-grained annotation of video datasets require exorbitant labeling costs and considerable time overhead. To address this challenge, we present DiffSynth-LVOS, a novel method that contains a diffusion-based object video generation model to create synthetic various training data according to textual object description. Our method first generates high-quality videos conditioned on masks from existing real datasets by introducing a full-fill mask constraint and video latent blending module, ensuring precise frame-mask alignment and temporal consistency for video segmentation training. Then the impact of synthetic videos through joint fine-tuning with real videos is investigated to increase the performance of LVOS. Experimental results demonstrate significant performance improvements when training on our expanded dataset (e.g., \( \mathcal {J} \& \mathcal {F}\) score improves from 52.39% to 56.48% on Ref-DAVIS benchmark). Furthermore, comparative studies reveal that our method produces object segmentation videos more suitable for LVOS tasks than alternative video generation approaches. This work provides an effective solution to the data scarcity problem in LVOS while maintaining model performance with reduced reliance on annotated real videos.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DiffSynth-LVOS: Enhancing Language-Guided Video Object Segmentation via Diffusion-Based Synthetic Data Generation

  • Chunjiang He,
  • Gang Yang

摘要

Language-guided video object segmentation (LVOS) has achieved remarkable progress through advanced video segmentation models, but their performance heavily relies on large-scale fine-grained training data. The acquisition and fine-grained annotation of video datasets require exorbitant labeling costs and considerable time overhead. To address this challenge, we present DiffSynth-LVOS, a novel method that contains a diffusion-based object video generation model to create synthetic various training data according to textual object description. Our method first generates high-quality videos conditioned on masks from existing real datasets by introducing a full-fill mask constraint and video latent blending module, ensuring precise frame-mask alignment and temporal consistency for video segmentation training. Then the impact of synthetic videos through joint fine-tuning with real videos is investigated to increase the performance of LVOS. Experimental results demonstrate significant performance improvements when training on our expanded dataset (e.g., \( \mathcal {J} \& \mathcal {F}\) score improves from 52.39% to 56.48% on Ref-DAVIS benchmark). Furthermore, comparative studies reveal that our method produces object segmentation videos more suitable for LVOS tasks than alternative video generation approaches. This work provides an effective solution to the data scarcity problem in LVOS while maintaining model performance with reduced reliance on annotated real videos.