Feature Fusion and Multi-head Attention Based Hindi Captioner

Deep learning-based methods are extensively used in image captioning, but most of these methods depend on features from a single encoder for generating captions. Different encoders capture different features of an image, and thus, using features from multiple encoders may help improve the models’ performance. Moreover, there needs to be more research on Hindi caption generation on large datasets such as MSCOCO. Recently, transformers have performed well on tasks such as image classification and object detection. One such transformer is the Swin Transformer. It captures both local as well as global information present in the image. A Faster RCNN, on the other hand, captures only local (object-level) information but does not capture global details. Using a single image feature generation method might sometimes result in incorrect feature generation, or some important objects may be missed while generating the feature vector. This problem can be mitigated by combining features from different methods. Furthermore, as local features-based models have produced better results in different domains, utilizing both Swin Transformer and Faster RCNN may result in better captioning models. This work proposes to use Swin Transformer-based image features along with Faster RCNN-based image features to generate Hindi captions for images. A decoder with two GRUs and Multi-head Attention uses these image features to build Hindi captions. Experiments demonstrate that the proposed method can generate high-quality captions while improving the performance of automatic evaluation metrics, establishing the method’s efficacy.