The CAA Captioner–Enhancing Image Captioning with Contrastive Learning and Attention on Attention Mechanism

Image Captioning is a complex interdisciplinary task, situated at the intersection of CV (computer vision) and NLP (natural language processing). In recent years, with the development of encoder decoder frameworks and attention mechanisms, although significant progress has been made in this task, there is still a need to optimize the accuracy and completeness of text generation to improve the generalization ability of the model. We propose an Image Captioning method that combines a contrastive learning-enhanced ResNet101 encoder with an Attention on Attention (AoA) improved Transformer decoder to achieve better performance in generating accurate and natural descriptions. We utilize contrastive learning to enhance the model's understanding of image content. Additionally, we apply data augmentation techniques to enhance the model's generalization ability, and optimize the attention mechanism of the Transformer decoder using the AoA module. Our experimental results demonstrate the effectiveness of contrastive learning in handling complex scenes, multiple objects, and detailed descriptions, as our approach achieves superior performance on widely adopted Image Captioning datasets.