The aim of the paper is to create and implement a deep-learning model which can automatically generate captions for images, without human involvement. This approach uses the most recent advancements in computer vision and natural language processing to relate the underlying patterns and relationships between textual and visual data. Our model is trained on a common dataset with additional captions that are distinct in their descriptiveness, in contrast to the transformer architecture. The final model can be used to modify the imageability and output lengths for different applications. In the proposed approach the model is trained using the dataset “Microsoft COCO”. The model is supplied with the sample images and their associated text, as well as puts a new test image to generate captions based on its training data. The MS-COCO (Common Objects in Context) dataset is a massive image recognition data set for tasks such as object detection, segmentation and captioning. It has more than 370,00 images in which each annotated with 81 object classes and five categories for the scene. The Vision Transformer operates as an encoder in this model. The input image is fed into ViT to derive features. Finally, the output from ViT’s last hidden state connects to decoder. The decoder is a BERT that generates the captions. After encoding and concatenation of the images with their associated captions using Vision Transformer, BERT model is used to decode these encoded elements.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Image Captioning Using ViT and BERT Models

  • J. Avanija,
  • Narahari Harshini,
  • R. Harsha Sree,
  • Madugundu Vasu,
  • E. Pavan Kalyan,
  • Viswaksena Balaji,
  • Sam Goundar

摘要

The aim of the paper is to create and implement a deep-learning model which can automatically generate captions for images, without human involvement. This approach uses the most recent advancements in computer vision and natural language processing to relate the underlying patterns and relationships between textual and visual data. Our model is trained on a common dataset with additional captions that are distinct in their descriptiveness, in contrast to the transformer architecture. The final model can be used to modify the imageability and output lengths for different applications. In the proposed approach the model is trained using the dataset “Microsoft COCO”. The model is supplied with the sample images and their associated text, as well as puts a new test image to generate captions based on its training data. The MS-COCO (Common Objects in Context) dataset is a massive image recognition data set for tasks such as object detection, segmentation and captioning. It has more than 370,00 images in which each annotated with 81 object classes and five categories for the scene. The Vision Transformer operates as an encoder in this model. The input image is fed into ViT to derive features. Finally, the output from ViT’s last hidden state connects to decoder. The decoder is a BERT that generates the captions. After encoding and concatenation of the images with their associated captions using Vision Transformer, BERT model is used to decode these encoded elements.