Video captioning aims to automatically generate textual descriptions of video content, enhancing accessibility, comprehension, and searchability of videos. Recent advancements in deep learning, particularly in object recognition and encoder-decoder architectures, have significantly propelled the field forward. However, existing models may generate incomplete or grammatically incorrect sentences, particularly when it comes to non-visual words such as prepositions and conjunctions. To alleviate this problem, we introduce PosCap, a part-of-speech (POS) assisted video captioning model. Leveraging POS information as the prior knowledge, PosCap enhances word prediction by focusing on distinctive multimodal features for different word generation. Specifically, we introduce a POS prediction module to predict the POS of the next word. The predicted POS information guides the attention mechanism to better integrate different modalities information, therefore effectively improving word generation. Experimental results on two benchmark datasets, convincingly demonstrate that PosCap outperforms existing methods in generating coherent and grammatically correct video descriptions.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

PosCap: Boosting Video Captioning with Part-of-Speech Guidance

  • Jingfu Xiao,
  • Zhiliang Chen,
  • Wenhui Jiang,
  • Yuming Fang,
  • Fei Shen

摘要

Video captioning aims to automatically generate textual descriptions of video content, enhancing accessibility, comprehension, and searchability of videos. Recent advancements in deep learning, particularly in object recognition and encoder-decoder architectures, have significantly propelled the field forward. However, existing models may generate incomplete or grammatically incorrect sentences, particularly when it comes to non-visual words such as prepositions and conjunctions. To alleviate this problem, we introduce PosCap, a part-of-speech (POS) assisted video captioning model. Leveraging POS information as the prior knowledge, PosCap enhances word prediction by focusing on distinctive multimodal features for different word generation. Specifically, we introduce a POS prediction module to predict the POS of the next word. The predicted POS information guides the attention mechanism to better integrate different modalities information, therefore effectively improving word generation. Experimental results on two benchmark datasets, convincingly demonstrate that PosCap outperforms existing methods in generating coherent and grammatically correct video descriptions.