<p>Video captioning is a very essential and vital activity of the field of artificial intelligence with a view to connect the visual image to the natural language processing. It actively contributes to accessibility, indexing of the media, and automatic video summarization. Most of these approaches employed in the past have focused on object identification and action recognition with little attention paid to the meaning of these relationships and temporal hints. The methodology of video captioning in this paper is much more advanced than the previous methods. Therefore, it to a great extent is contingent upon multimodal learning methods that enhance precision and contextuality in generating captions. The majority of the past methods have extensively focused on object and action recognition and have omitted entirely the dynamism of video sequences, and in this regard, our novel technology will combine scene knowledge with BLIP, action recognition with CLIP, and through Google Speech Transcription as an audio-related attribute. Also, our solution expands video captioning to facilitate the use of multiple languages, which will make it accessible to a broader audience. The captions are automatically obtained in various languages, and the audio output is offered to obtain a more in-depth interpretation. These modalities are therefore incorporated in our approach and a ranking system of CLIP scores is added to produce highly relevant and semantically enriched captions. We show that the model is substantially superior to traditional methods in terms of the quality, expressiveness and applicability of the generated captions. The significance of the research is the development of the automated video interpretation.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Ensemble-based multilingual video captioning with multimodel fusion of visual and audio cues

  • Lakshmi Harika Palivela,
  • Sandip Datta,
  • Suri Venkata Rohith Kumar,
  • Doradla Pardha Saradhi Raju

摘要

Video captioning is a very essential and vital activity of the field of artificial intelligence with a view to connect the visual image to the natural language processing. It actively contributes to accessibility, indexing of the media, and automatic video summarization. Most of these approaches employed in the past have focused on object identification and action recognition with little attention paid to the meaning of these relationships and temporal hints. The methodology of video captioning in this paper is much more advanced than the previous methods. Therefore, it to a great extent is contingent upon multimodal learning methods that enhance precision and contextuality in generating captions. The majority of the past methods have extensively focused on object and action recognition and have omitted entirely the dynamism of video sequences, and in this regard, our novel technology will combine scene knowledge with BLIP, action recognition with CLIP, and through Google Speech Transcription as an audio-related attribute. Also, our solution expands video captioning to facilitate the use of multiple languages, which will make it accessible to a broader audience. The captions are automatically obtained in various languages, and the audio output is offered to obtain a more in-depth interpretation. These modalities are therefore incorporated in our approach and a ranking system of CLIP scores is added to produce highly relevant and semantically enriched captions. We show that the model is substantially superior to traditional methods in terms of the quality, expressiveness and applicability of the generated captions. The significance of the research is the development of the automated video interpretation.