<p>Vehicle counting is a critical task in traffic monitoring and management. Although large vision-language models have achieved significant progress in zero-shot text-image matching, adapting them for vehicle counting remains a challenging task. To tackle this issue, we propose FCLIP-VC, a Fourier-modulated Contrastive Language-image Pre-training (CLIP) for zero-shot vehicle counting. First, a Vision Transformer (ViT)-based CLIP model is employed to encode image features for precise vehicle localization. Additionally, we introduce a learnable visual prompt to avoid tuning global parameters, allowing the pre-trained CLIP knowledge to be effectively applied to dense prediction tasks. Next, for patch-level embedding of images, we introduce a patch-language contrast loss to optimize the effectiveness of the model in capturing fine-grained features of images. Then, we propose a patch-language Discrete Fourier transform (DFT) interaction module, leveraging DFT to extract multi-scale image features in the frequency domain, which improves the ability of the model to handle diverse vehicle sizes and complex traffic environments. Finally, a content-aware density map decoder is designed to generate accurate density map predictions through multi-layer convolution and progressive upsampling. Extensive experiments demonstrate that our proposed FCLIP-VC achieves state-of-the-art accuracy in zero-shot vehicle counting.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Fourier-modulated CLIP for zero-shot vehicle counting

  • Yunpeng Luo,
  • Qin Zhang,
  • Jiawei Luo,
  • Shixin Huang,
  • Xiaoyu Wan,
  • Xixi Nie

摘要

Vehicle counting is a critical task in traffic monitoring and management. Although large vision-language models have achieved significant progress in zero-shot text-image matching, adapting them for vehicle counting remains a challenging task. To tackle this issue, we propose FCLIP-VC, a Fourier-modulated Contrastive Language-image Pre-training (CLIP) for zero-shot vehicle counting. First, a Vision Transformer (ViT)-based CLIP model is employed to encode image features for precise vehicle localization. Additionally, we introduce a learnable visual prompt to avoid tuning global parameters, allowing the pre-trained CLIP knowledge to be effectively applied to dense prediction tasks. Next, for patch-level embedding of images, we introduce a patch-language contrast loss to optimize the effectiveness of the model in capturing fine-grained features of images. Then, we propose a patch-language Discrete Fourier transform (DFT) interaction module, leveraging DFT to extract multi-scale image features in the frequency domain, which improves the ability of the model to handle diverse vehicle sizes and complex traffic environments. Finally, a content-aware density map decoder is designed to generate accurate density map predictions through multi-layer convolution and progressive upsampling. Extensive experiments demonstrate that our proposed FCLIP-VC achieves state-of-the-art accuracy in zero-shot vehicle counting.