Scholarlink

<p>Image captioning aims to generate coherent and natural language descriptions from images, requiring a profound understanding of both visual and semantic information. Despite recent advancements, existing attention-based methods for image captioning still face challenges in effectively leveraging global semantic information and positional relationships between image elements. In this paper, we propose the Semantically Enhanced Dual Visual Fusion Transformer (SEDVFT), which integrates region and grid visual features with semantic information to improve caption quality. The core of our approach is the Semantically Guided Visual Fusion Attention (SGVFA) module, which enhances the granularity of visual features by incorporating semantic information and facilitating interaction between region and grid features. Furthermore, we introduce the Position Enhanced Self-Attention (PESA) module, which utilizes both absolute and relative positional encodings to capture geometric correlations among visual instances. Experimental results on the MSCOCO dataset demonstrate that our SEDVFT model achieves strong performance, with a CIDEr score of 135.4% on the Karpathy split, highlighting the effectiveness of our proposed approach in integrating and utilizing multi-faceted visual and semantic cues for accurate image captioning. The code is available at <a href="https://github.com/Zzzzz1l/SEDVFT">https://github.com/Zzzzz1l/SEDVFT</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Semantically Enhanced Dual Visual Fusion Transformer for accurate image captioning

  • 期刊论文

  • 发表日期：

    2025年5月8日

查看全文
  • Juan Yang,
  • Hao Zhang,
  • Ronggui Wang,
  • Lixia Xue,
  • Jiaping Zhang

摘要

Image captioning aims to generate coherent and natural language descriptions from images, requiring a profound understanding of both visual and semantic information. Despite recent advancements, existing attention-based methods for image captioning still face challenges in effectively leveraging global semantic information and positional relationships between image elements. In this paper, we propose the Semantically Enhanced Dual Visual Fusion Transformer (SEDVFT), which integrates region and grid visual features with semantic information to improve caption quality. The core of our approach is the Semantically Guided Visual Fusion Attention (SGVFA) module, which enhances the granularity of visual features by incorporating semantic information and facilitating interaction between region and grid features. Furthermore, we introduce the Position Enhanced Self-Attention (PESA) module, which utilizes both absolute and relative positional encodings to capture geometric correlations among visual instances. Experimental results on the MSCOCO dataset demonstrate that our SEDVFT model achieves strong performance, with a CIDEr score of 135.4% on the Karpathy split, highlighting the effectiveness of our proposed approach in integrating and utilizing multi-faceted visual and semantic cues for accurate image captioning. The code is available at https://github.com/Zzzzz1l/SEDVFT.