Research on image caption generation method based on multi-modal pre-training model and text mixup optimization

In recent years, multi-modal pre-training models have demonstrated remarkable cross-modal representation capabilities, catalyzing the rapid evolution of multi-modal downstream tasks, particularly in image caption generation, towards a more comprehensive end-to-end framework. This evolutionary process has significantly enhanced the performance of such models. However, current technology has yet to fully integrate multi-modal information related to the scene into the end-to-end output. Consequently, the model's accuracy in object description is deficient, and its generalization ability across different scenes remains inadequate. To address these challenges, this paper presents the MTMixIC model, an innovative solution for image caption generation. The MTMixIC model adopts a multi-modal pre-training approach, integrating text mixup optimization techniques to enhance image caption generation. Leveraging the intricate relationship between images and text, it directly translates images into descriptive text, even in zero-shot scenarios, achieving end-to-end caption generation. This addresses the limitations of single-modal pre-training models, which struggle with multi-modal characteristics, resulting in improved caption quality. Additionally, a text mixup optimization network refines generated captions by exploiting correlations between multi-modal features and captions. This not only maintains semantic accuracy but also aligns better with the annotation style of the scene, addressing suboptimal performance in zero-shot caption generation and enhancing generalization capabilities. Experimental validation on datasets like MS COCO14, MS COCO17, Flickr8k, Sydney, RSICD, etc., demonstrates superior accuracy, fluency, and generalization compared to traditional architectures. The proposed model offers a novel approach, contributing to the evolution of multi-modal downstream tasks.