VGGIN-Net–Transformer: Multimodal Encoder-Decoder Framework for Semantic Image Captioning
摘要
This paper emphasizes on developing a multimodal image captioning system to generate accurate semantic image descriptions through the integration of advanced deep learning models with tailored architectures. The system translates low-level visual cues to high-level text information via a novel multimodal encoder-decoder framework for generating captions that depict image contents alongside their situational context. Specifically, the approach integrates VGGIN-Net, a recently introduced deep pre-trained convolutional neural network (CNN) model, with a Transformer, for enhanced image-to-text translation. The VGGIN-Net architecture combines the lower bottleneck layers of VGG with a randomly initialized Inception module; this combination was proved highly effective for imbalanced datasets. The top layer of VGGIN-Net comprising of flattened features is given as input to a Transformer decoder that generates a sequence of words corresponding to the image description. Among all deep learning models investigated in this study, the proposed combination of VGGIN-Net with Transformer achieves the highest BLEU-1 (0.6976), BLEU-2 (0.5378), BLEU-3 (0.4020), BLEU-4 (0.2340), METEOR (0.221), and CIDEr (0.477) scores on the benchmark Flickr8k dataset; these metrics measure the similarity between the machine-generated image captions and human-written reference texts. The proposed VGGIN-Net–Transformer encoder-decoder framework boosts the performance of image captioning systems through improved accuracy and reliability by handling problems of overfitting as well as semantic errors and model limitations in dataset generalization.