Scholarlink

<p>Visual grounding is a fundamental task that bridges vision and language, aiming to accurately associate natural language queries with specific regions in an image. Existing approaches, predominantly based on Transformers or CNNs, struggle with balancing computational efficiency and fine-grained semantic alignment. In this paper, we propose RWKV-VG, the first visual grounding framework entirely built on the RWKV architecture. Leveraging RWKV’s unique ability to combine RNN-like sequential modeling with Transformer-like attention, our model efficiently achieves both intra-modal and cross-modal reasoning. The framework consists of a RWKV-based visual encoder, a RWKV-based linguistic encoder, and a RWKV-based visual-linguistic decoder, complemented by a learnable [REG] token designed for box regression. Comprehensive evaluations on benchmark datasets, including ReferItGame and the RefCOCO series, demonstrate the superiority of RWKV-VG, achieving state-of-the-art performance with rapid convergence. Ablation studies further confirm the effectiveness of the RWKV modules and the [REG] token design. Our work establishes RWKV as a compelling alternative to conventional architectures for visual grounding tasks. To facilitate future research, the code and pre-trained models are released at <a href="https://github.com/nianfd/RWKV-VG">https://github.com/nianfd/RWKV-VG</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Rwkv-vg: visual grounding with RWKV-driven encoder-decoder framework

  • 期刊论文

  • 发表日期：

    2025年2月21日

查看全文
  • Fudong Nian,
  • Yanhong Gu,
  • Wentao Wang,
  • Aoyu Liu,
  • Dong Zhang,
  • Fanding Li

摘要

Visual grounding is a fundamental task that bridges vision and language, aiming to accurately associate natural language queries with specific regions in an image. Existing approaches, predominantly based on Transformers or CNNs, struggle with balancing computational efficiency and fine-grained semantic alignment. In this paper, we propose RWKV-VG, the first visual grounding framework entirely built on the RWKV architecture. Leveraging RWKV’s unique ability to combine RNN-like sequential modeling with Transformer-like attention, our model efficiently achieves both intra-modal and cross-modal reasoning. The framework consists of a RWKV-based visual encoder, a RWKV-based linguistic encoder, and a RWKV-based visual-linguistic decoder, complemented by a learnable [REG] token designed for box regression. Comprehensive evaluations on benchmark datasets, including ReferItGame and the RefCOCO series, demonstrate the superiority of RWKV-VG, achieving state-of-the-art performance with rapid convergence. Ablation studies further confirm the effectiveness of the RWKV modules and the [REG] token design. Our work establishes RWKV as a compelling alternative to conventional architectures for visual grounding tasks. To facilitate future research, the code and pre-trained models are released at https://github.com/nianfd/RWKV-VG.