Video captioning stands at the intersection of computer vision and natural language processing, presenting unique challenges in interpreting dynamic visual content and generating coherent textual descriptions. Traditional approaches often struggle with capturing the temporal intricacies and complex interactions inherent in video data. This paper introduces the (STG-VC), a novel framework designed to enhance the accuracy and contextuality of video captions. By representing videos as spatio-temporal graphs, the STG-VC model captures detailed interactions and temporal dynamics that conventional sequence-based methods often overlook. The core of the model consists of a graph transformer encoder that processes these graphs to produce rich contextual embeddings. A sophisticated caption generation decoder then translates these embeddings into descriptive, accurate captions. We evaluate the STG-VC model across several challenging datasets, including YouCook2, Flickr30k, and Activity Net, where it demonstrates superior performance in generating contextually relevant and temporally coherent captions compared to existing state-of-the-art methods. This work not only advances the technology of video captioning but also opens new avenues for future research in multimedia content analysis.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Video Captioning with Spatio-Temporal Graph Transformers

  • Shakhnoza Muksimova,
  • Sabina Umirzakova,
  • Sevara Mardieva,
  • Nargiza Iskhakova,
  • Young Im Cho

摘要

Video captioning stands at the intersection of computer vision and natural language processing, presenting unique challenges in interpreting dynamic visual content and generating coherent textual descriptions. Traditional approaches often struggle with capturing the temporal intricacies and complex interactions inherent in video data. This paper introduces the (STG-VC), a novel framework designed to enhance the accuracy and contextuality of video captions. By representing videos as spatio-temporal graphs, the STG-VC model captures detailed interactions and temporal dynamics that conventional sequence-based methods often overlook. The core of the model consists of a graph transformer encoder that processes these graphs to produce rich contextual embeddings. A sophisticated caption generation decoder then translates these embeddings into descriptive, accurate captions. We evaluate the STG-VC model across several challenging datasets, including YouCook2, Flickr30k, and Activity Net, where it demonstrates superior performance in generating contextually relevant and temporally coherent captions compared to existing state-of-the-art methods. This work not only advances the technology of video captioning but also opens new avenues for future research in multimedia content analysis.