Scholarlink

<p>In this work, we tackle the fundamental multimodal task of image-text retrieval, where fine-grained alignment between elements such as image patches and texts is the key to achieving effective retrieval. The alignment process consists of two stages: learning unified semantics and matching similar elements. To overcome the challenges of fragment-based fusion, redundant matches, and multimodal inconsistency in these two stages, we propose a Graph-based Fine-Grained multimodal Alignment (GFGA) method. Specifically, we adopt a concept-based fusion module in stage&#xa0;I to complement typical fragment-based fusion, which lacks comprehensive interactions. Furthermore, we introduce a node masker in stage&#xa0;II to mask unnecessary elements and mitigate the problem of redundant matching. Finally, an inconsistency-aware graph matching module simultaneously matches consistent elements and captures multimodal inconsistency. By integrating these modules, our method can overcome the challenges of current methods and enhance both stages for fine-grained multimodal alignment. The extensive experiments on two benchmark datasets demonstrate the effectiveness of our method and its superior performance in image-text retrieval. The source codes are available at <a href="https://github.com/maochen-casia/gfga">https://github.com/maochen-casia/gfga</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Fine-Grained Multimodal Alignment for Image-Text Retrieval via Graph Learning

  • 期刊论文

  • 发表日期：

    2026年2月21日

查看全文
  • Mao Chen,
  • Xiangkai Zhang,
  • Lu Qi,
  • Xiangtai Li,
  • Xu Yang,
  • Steven C. H. Hoi,
  • Zhiyong Liu,
  • Ming-Hsuan Yang

摘要

In this work, we tackle the fundamental multimodal task of image-text retrieval, where fine-grained alignment between elements such as image patches and texts is the key to achieving effective retrieval. The alignment process consists of two stages: learning unified semantics and matching similar elements. To overcome the challenges of fragment-based fusion, redundant matches, and multimodal inconsistency in these two stages, we propose a Graph-based Fine-Grained multimodal Alignment (GFGA) method. Specifically, we adopt a concept-based fusion module in stage I to complement typical fragment-based fusion, which lacks comprehensive interactions. Furthermore, we introduce a node masker in stage II to mask unnecessary elements and mitigate the problem of redundant matching. Finally, an inconsistency-aware graph matching module simultaneously matches consistent elements and captures multimodal inconsistency. By integrating these modules, our method can overcome the challenges of current methods and enhance both stages for fine-grained multimodal alignment. The extensive experiments on two benchmark datasets demonstrate the effectiveness of our method and its superior performance in image-text retrieval. The source codes are available at https://github.com/maochen-casia/gfga.