Scholarlink

<p>Image captioning, a process that integrates computer vision and natural language processing, aims to transform the visual content into a descriptive text representation. Despite significant progress, accurately capturing complex object relationships and interactions within images remains challenging. Scene graphs, which model objects and their pairwise relationships, have proven to be effective in enhancing captioning systems by providing structured semantic representations. However, reasoning about intricate relationships within these graphs continues to limit the quality and depth of generated captions. Building on our previous work in scene graph generation, which introduced a human-inspired approach leveraging semantic spatial relationships and a message-passing mechanism to model object interactions, we now extend this work to integrate image captioning. This end-to-end system not only generates scene graphs but also translates them into detailed and contextually rich textual descriptions. A key innovation is incorporating large language models (LLMs) to interpret scene graph structures and produce high-quality captions encompassing complex actions and relationships. We validate our method through extensive experiments on the Visual Genome and MSCOCO datasets. The results demonstrate the system’s robustness, even with a limited scene graph vocabulary, highlighting its ability to handle scenarios with sparse or incomplete scene information.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Boosting scene captioning with pairwise semantic spatial reasoning and contextualized large language model integration

  • 期刊论文

  • 发表日期：

    2026年8月5日

查看全文
  • Anfel Amirat,
  • Nadia Baha,
  • Lamine Benrais

摘要

Image captioning, a process that integrates computer vision and natural language processing, aims to transform the visual content into a descriptive text representation. Despite significant progress, accurately capturing complex object relationships and interactions within images remains challenging. Scene graphs, which model objects and their pairwise relationships, have proven to be effective in enhancing captioning systems by providing structured semantic representations. However, reasoning about intricate relationships within these graphs continues to limit the quality and depth of generated captions. Building on our previous work in scene graph generation, which introduced a human-inspired approach leveraging semantic spatial relationships and a message-passing mechanism to model object interactions, we now extend this work to integrate image captioning. This end-to-end system not only generates scene graphs but also translates them into detailed and contextually rich textual descriptions. A key innovation is incorporating large language models (LLMs) to interpret scene graph structures and produce high-quality captions encompassing complex actions and relationships. We validate our method through extensive experiments on the Visual Genome and MSCOCO datasets. The results demonstrate the system’s robustness, even with a limited scene graph vocabulary, highlighting its ability to handle scenarios with sparse or incomplete scene information.