Scholarlink

<p>Image captioning is a fundamental task in the multimodal field, where the goal is to transform images into coherent text through deep network processing. Simple grid features often exhibit subpar performance due to their lack of contextual information and the presence of excessive noise. This paper aims to address these issues by enhancing fine-grained grid features with contextual semantic information within the original transformer model framework. We proposed a novel Dilated Attention Fusion Transformer (DAFT). Firstly, we integrate semantic segmentation features through a Feature Fusion Module Based on Cross-Attention to capture object-related information comprehensively. We then propose a novel multi-scale multi-head sparse attention mechanism based on grids, which improves granularity while reducing unnecessary noise and computational costs. Additionally, we employ a weighted residual connection method to fuse multi-layer information, generating richer representations. Extensive experiments on MS-COCO dataset demonstrate the effectiveness of our DAFT, with improvements of CIDEr from 133.2 to 135.7%, achieving significantly improved performance over the baseline. The source code is available at <a href="https://github.com/lishuo19981027/DAFT.">https://github.com/lishuo19981027/DAFT.</a></p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Enhancing visual contextual semantic information for image captioning

  • 期刊论文

  • 发表日期：

    2025年5月11日

查看全文
  • Ronggui Wang,
  • Shuo Li,
  • Lixia Xue,
  • Juan Yang

摘要

Image captioning is a fundamental task in the multimodal field, where the goal is to transform images into coherent text through deep network processing. Simple grid features often exhibit subpar performance due to their lack of contextual information and the presence of excessive noise. This paper aims to address these issues by enhancing fine-grained grid features with contextual semantic information within the original transformer model framework. We proposed a novel Dilated Attention Fusion Transformer (DAFT). Firstly, we integrate semantic segmentation features through a Feature Fusion Module Based on Cross-Attention to capture object-related information comprehensively. We then propose a novel multi-scale multi-head sparse attention mechanism based on grids, which improves granularity while reducing unnecessary noise and computational costs. Additionally, we employ a weighted residual connection method to fuse multi-layer information, generating richer representations. Extensive experiments on MS-COCO dataset demonstrate the effectiveness of our DAFT, with improvements of CIDEr from 133.2 to 135.7%, achieving significantly improved performance over the baseline. The source code is available at https://github.com/lishuo19981027/DAFT.