Context-Aware Image Description via Dual-Stream Visual Encoding and Guided Multimodal Learning
摘要
Appropriate representation of image content remains a core challenge in describing images. While numerous methods have been developed, many rely solely on full image features, which often include irrelevant background information. Others focus on object regions, potentially neglecting crucial contextual cues. To overcome these limitations, the proposed framework combines object-region features with entire image representations. It employs a Faster R-CNN to extract discriminative object features, fused with holistic scene context from ConvNeXt and fed into an LSTM-guided Transformer decoder for improved modelling of sequential dependencies. Extensive evaluations on the MS COCO dataset demonstrate the effectiveness of the proposed approach, achieving a BLEU-4 score of 0.397, METEOR of 0.298, and CIDEr of 1.342, outperforming several state-of-the-art models in generating context-aware, rich image descriptions.