In recent years, more researchers in the field of multi-modal tracking have focused on various algorithms for RGB-T tracking, leveraging the complementary nature of RGB and TIR imaging to achieve good application results. However, their performance tends to degrade in specific scenarios where the target is color-camouflaged and TIR modality is ineffective. Our experiments reveal that ultraviolet (UV) sensors can effectively image certain camouflage materials. Therefore, in extreme scenarios where the target is color-camouflaged and TIR modality is ineffective, UV modality can serve as a supplementary means to RGB and TIR modalities, enhancing tracking performance. In this paper, we propose the first multi-modal object tracking network for visible light, thermal infrared, and ultraviolet, namely VTUTrack, which achieves better tracking performance in complex scenarios. Furthermore, to meet the needs of real-time tracking applications, we introduce an adaptive candidate elimination mechanism based on modality reliability within the ViT (Vision Transformer) backbone network, reducing the computational burden of multi-modal feature extraction and improving tracking inference speed. Extensive experiments further demonstrate the effectiveness of our proposed RGB-T-UV multi-modal object tracking method.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

RGB-T-UV Multi-modal Object Tracking Based on Transformer Network

  • Qinghua Song,
  • Xiaolei Wang,
  • Yi Zhang,
  • Jinping Hu,
  • Yu Liu

摘要

In recent years, more researchers in the field of multi-modal tracking have focused on various algorithms for RGB-T tracking, leveraging the complementary nature of RGB and TIR imaging to achieve good application results. However, their performance tends to degrade in specific scenarios where the target is color-camouflaged and TIR modality is ineffective. Our experiments reveal that ultraviolet (UV) sensors can effectively image certain camouflage materials. Therefore, in extreme scenarios where the target is color-camouflaged and TIR modality is ineffective, UV modality can serve as a supplementary means to RGB and TIR modalities, enhancing tracking performance. In this paper, we propose the first multi-modal object tracking network for visible light, thermal infrared, and ultraviolet, namely VTUTrack, which achieves better tracking performance in complex scenarios. Furthermore, to meet the needs of real-time tracking applications, we introduce an adaptive candidate elimination mechanism based on modality reliability within the ViT (Vision Transformer) backbone network, reducing the computational burden of multi-modal feature extraction and improving tracking inference speed. Extensive experiments further demonstrate the effectiveness of our proposed RGB-T-UV multi-modal object tracking method.