UPVIS: upsampled video query for offline video instance segmentation

This paper presents a new approach for offline video instance segmentation (VIS) by integrating upsampling layers into the Transformer decoder. Existing offline VIS techniques set the temporal dimension of the video query as either one or the length of the input video, which poses challenges for tracking due to limited query capacity in the former and implementation issues in the latter. To overcome these challenges, we introduce upsampling of video queries along the temporal axis during the decoding process. This eases the induction of association property since the same instance information is expanded along the temporal axis, considering temporal adjacency. Additionally, by representing an instance with a diverse set of queries, more refined predictions become possible. We also address the ambiguity that arises when the input video length is not divisible by the video queries’ temporal length by interpolating the resultant video queries to match the video sequences. To the best of our knowledge, this is the first paper that handles video queries with arbitrary temporal dimensions, neither one nor the length of the input video. We have conducted extensive experiments to validate the effectiveness of the proposed method in the offline scheme on YouTubeVIS-2019/2021/2022, and OVIS benchmarks. In particular, our proposed method exhibited notable performance in handling long and challenging videos.