Appropriate representation of image content remains a core challenge in describing images. While numerous methods have been developed, many rely solely on full image features, which often include irrelevant background information. Others focus on object regions, potentially neglecting crucial contextual cues. To overcome these limitations, the proposed framework combines object-region features with entire image representations. It employs a Faster R-CNN to extract discriminative object features, fused with holistic scene context from ConvNeXt and fed into an LSTM-guided Transformer decoder for improved modelling of sequential dependencies. Extensive evaluations on the MS COCO dataset demonstrate the effectiveness of the proposed approach, achieving a BLEU-4 score of 0.397, METEOR of 0.298, and CIDEr of 1.342, outperforming several state-of-the-art models in generating context-aware, rich image descriptions.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Context-Aware Image Description via Dual-Stream Visual Encoding and Guided Multimodal Learning

  • Biswajit Patra,
  • Chiradeep Ghosh,
  • Dakshina Ranjan Kisku

摘要

Appropriate representation of image content remains a core challenge in describing images. While numerous methods have been developed, many rely solely on full image features, which often include irrelevant background information. Others focus on object regions, potentially neglecting crucial contextual cues. To overcome these limitations, the proposed framework combines object-region features with entire image representations. It employs a Faster R-CNN to extract discriminative object features, fused with holistic scene context from ConvNeXt and fed into an LSTM-guided Transformer decoder for improved modelling of sequential dependencies. Extensive evaluations on the MS COCO dataset demonstrate the effectiveness of the proposed approach, achieving a BLEU-4 score of 0.397, METEOR of 0.298, and CIDEr of 1.342, outperforming several state-of-the-art models in generating context-aware, rich image descriptions.