Aware-Transformer: A Novel Pure Transformer-Based Model for Remote Sensing Image Captioning

Remote sensing image captioning (RSIC) is the task of generating accurate and coherent descriptions of the visual content in remote sensing images. While recent progress has been made in developing CNN-Transformer based models for this task, given the significant scale differences in the visual objects within these images, many existing methods still have some deficiencies in effectively capturing the multiscale visual features of these images. Additionally, applying these visual features directly to a vanilla Transformer architecture may result in the loss of important visual information. To address these challenges, we propose a novel pure Transformer-based model that first utilizes a fine-tuned Swin-Transformer as the encoder to extract multiscale visual features from remote sensing images. Then it introduces an Aware-Transformer as the decoder, which enhances multiscale and multiobject visual information to help generate accurate and detailed captions. To assess the performance of our proposed method, we conducted ablation and comparison experiments on three publicly available RSIC datasets: Sydney-Captions, UCM-Captions, and NWPU-Captions. The results demonstrate that our method outperforms state-of-the-art RSIC models in captioning quality.