Multilayer Vision and Language Augmented Transformer for Image Captioning

Image Captioning task is one of the important tasks in computer vision. In this paper, we propose a Multilayer Vision and Language Augmented Transformer (MVLAT) method for improving the correctness of image description statements. In MVLAT, we have matched image and text features by adding regional object image features as well as grid image features and introduced them into the image description task by cross-modal retrieval. We refer to this process as the visual contextual relationship module (VRM) to enhance the visual and contextual features of the image description model. In addition, to focus attention more on the focused information, we propose an attention enhancement module (AEM) that enhances the attention weight of important information while weakening the attention weight of non-essential information. Finally, we propose a multi-label Focal loss in the image description task that balances the positive and negative sample imbalance in the training model. Experiments on the MSCOCO image description baseline dataset show that the present method can obtain good performance, and the overall performance of the model is better than many existing state-of-the-art methods. The improvement over the baseline model is 7.7 on the CIDEr score and 1.5 on the BLEU-1 score.