RGFormer: Residual Gated Transformer for Image Captioning

Image captioning is a cross-modal task that combines computer vision and natural language processing. The model is required to generate an appropriate caption for the given image. To address this challenge, we proposed a Residual Gated Transformer, RGFormer, as an enhancement of Transformer architecture. The model based on CLIP and RGFormer, CRM, is then proposed for image captioning. CRM utilizes the encoder of CLIP to extract image features as a prefix to the caption. The prefix is projected into language space using RGFormer, a lightweight mapping network, and then fed into GPT-2 to generate captions. CLIP was trained on an extensive dataset comprising image-text pairs, which contains rich visual and semantic information and is exceptionally well-suited for vision-language tasks. The core idea of CRM is to reduce the disparity between visual and textual representations by using RGFormer to accomplish the cross-modal task. CRM could generate meaningful captions for diverse and large-scale datasets in a short training time without additional annotations or pre-training. Quantitative evaluation experiments show that CRM achieves results comparable to some advanced models on the COCO Caption dataset more efficiently.