<p>Image-text matching is a fundamental task in vision and language research, crucial for bridging the semantic gap between visual and textual modalities. However, existing methods often struggle with irrelevant information interfering with precise fine-grained matching and frequently overlook potential positive samples. To address these critical issues, we propose global and local label-constrained alignment, a novel image-text matching framework that leverages both local and global semantic labels to guide fine-grained cross-modal alignment. For local labels, label cyclic matching consistency ensures consistent bidirectional alignment during text-image-text matching loops, suppressing background clutter. This component compels the model to learn highly precise fine-grained correspondences. For global labels, similarity transformation hard negative mining converts intra-modal similarities into a learnable offset that mines hard negatives while protecting latent positive samples. The hard negative samples, coupled with the global-local feature correspondence method, enable our model to differentiate easily confused samples more effectively. Comprehensive experiments on two public datasets demonstrate the superiority of the proposed method in both image-to-text and text-to-image retrieval scenarios.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Global and local label-constrained alignment for image-text matching

  • Yifang Niu,
  • Jun Huang,
  • Shuzhen Rao

摘要

Image-text matching is a fundamental task in vision and language research, crucial for bridging the semantic gap between visual and textual modalities. However, existing methods often struggle with irrelevant information interfering with precise fine-grained matching and frequently overlook potential positive samples. To address these critical issues, we propose global and local label-constrained alignment, a novel image-text matching framework that leverages both local and global semantic labels to guide fine-grained cross-modal alignment. For local labels, label cyclic matching consistency ensures consistent bidirectional alignment during text-image-text matching loops, suppressing background clutter. This component compels the model to learn highly precise fine-grained correspondences. For global labels, similarity transformation hard negative mining converts intra-modal similarities into a learnable offset that mines hard negatives while protecting latent positive samples. The hard negative samples, coupled with the global-local feature correspondence method, enable our model to differentiate easily confused samples more effectively. Comprehensive experiments on two public datasets demonstrate the superiority of the proposed method in both image-to-text and text-to-image retrieval scenarios.