This paper emphasizes on developing a multimodal image captioning system to generate accurate semantic image descriptions through the integration of advanced deep learning models with tailored architectures. The system translates low-level visual cues to high-level text information via a novel multimodal encoder-decoder framework for generating captions that depict image contents alongside their situational context. Specifically, the approach integrates VGGIN-Net, a recently introduced deep pre-trained convolutional neural network (CNN) model, with a Transformer, for enhanced image-to-text translation. The VGGIN-Net architecture combines the lower bottleneck layers of VGG with a randomly initialized Inception module; this combination was proved highly effective for imbalanced datasets. The top layer of VGGIN-Net comprising of flattened features is given as input to a Transformer decoder that generates a sequence of words corresponding to the image description. Among all deep learning models investigated in this study, the proposed combination of VGGIN-Net with Transformer achieves the highest BLEU-1 (0.6976), BLEU-2 (0.5378), BLEU-3 (0.4020), BLEU-4 (0.2340), METEOR (0.221), and CIDEr (0.477) scores on the benchmark Flickr8k dataset; these metrics measure the similarity between the machine-generated image captions and human-written reference texts. The proposed VGGIN-Net–Transformer encoder-decoder framework boosts the performance of image captioning systems through improved accuracy and reliability by handling problems of overfitting as well as semantic errors and model limitations in dataset generalization.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

VGGIN-Net–Transformer: Multimodal Encoder-Decoder Framework for Semantic Image Captioning

  • Akash Singh,
  • Ashu Dhama,
  • Aditya Lama,
  • Seba Susan

摘要

This paper emphasizes on developing a multimodal image captioning system to generate accurate semantic image descriptions through the integration of advanced deep learning models with tailored architectures. The system translates low-level visual cues to high-level text information via a novel multimodal encoder-decoder framework for generating captions that depict image contents alongside their situational context. Specifically, the approach integrates VGGIN-Net, a recently introduced deep pre-trained convolutional neural network (CNN) model, with a Transformer, for enhanced image-to-text translation. The VGGIN-Net architecture combines the lower bottleneck layers of VGG with a randomly initialized Inception module; this combination was proved highly effective for imbalanced datasets. The top layer of VGGIN-Net comprising of flattened features is given as input to a Transformer decoder that generates a sequence of words corresponding to the image description. Among all deep learning models investigated in this study, the proposed combination of VGGIN-Net with Transformer achieves the highest BLEU-1 (0.6976), BLEU-2 (0.5378), BLEU-3 (0.4020), BLEU-4 (0.2340), METEOR (0.221), and CIDEr (0.477) scores on the benchmark Flickr8k dataset; these metrics measure the similarity between the machine-generated image captions and human-written reference texts. The proposed VGGIN-Net–Transformer encoder-decoder framework boosts the performance of image captioning systems through improved accuracy and reliability by handling problems of overfitting as well as semantic errors and model limitations in dataset generalization.