Video Instance Segmentation (VIS) is a pivotal technology for various applications including autonomous driving and video editing. While existing approaches primarily focus on enhancing the accuracy on datasets, they often neglect the real-time performance of segmentation results. Current online video instance segmentation models can sequentially output instance segmentation results for each frame and associate instances across adjacent frames; however, their inference speed limits practical applications. Additionally, these models require substantial memory during training and inference, which hinders deployment. To address these issues, we introduce the RT-VIS model, which adopts a decoupled strategy for VIS, allowing the model to be divided into segmenter and tracker components, each trained independently. To enhance inference speed, we developed a new light-weight instance segmentation model and employed a novel tracker to facilitate inter-frame instance association. Our model balances inference speed and accuracy, achieving 22.6 FPS and 42.2 AP on the YouTube-VIS2019 dataset, with memory requirements during training and inference approximately half of the previous methods. The code is available at https://github.com/STOVAGtz/RT-VIS .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

RT-VIS: Real-Time Video Instance Segmentation with Light-Weight Decoupled Framework

  • Tianze Cao,
  • Sanyuan Zhao

摘要

Video Instance Segmentation (VIS) is a pivotal technology for various applications including autonomous driving and video editing. While existing approaches primarily focus on enhancing the accuracy on datasets, they often neglect the real-time performance of segmentation results. Current online video instance segmentation models can sequentially output instance segmentation results for each frame and associate instances across adjacent frames; however, their inference speed limits practical applications. Additionally, these models require substantial memory during training and inference, which hinders deployment. To address these issues, we introduce the RT-VIS model, which adopts a decoupled strategy for VIS, allowing the model to be divided into segmenter and tracker components, each trained independently. To enhance inference speed, we developed a new light-weight instance segmentation model and employed a novel tracker to facilitate inter-frame instance association. Our model balances inference speed and accuracy, achieving 22.6 FPS and 42.2 AP on the YouTube-VIS2019 dataset, with memory requirements during training and inference approximately half of the previous methods. The code is available at https://github.com/STOVAGtz/RT-VIS .