PosCap: Boosting Video Captioning with Part-of-Speech Guidance
摘要
Video captioning aims to automatically generate textual descriptions of video content, enhancing accessibility, comprehension, and searchability of videos. Recent advancements in deep learning, particularly in object recognition and encoder-decoder architectures, have significantly propelled the field forward. However, existing models may generate incomplete or grammatically incorrect sentences, particularly when it comes to non-visual words such as prepositions and conjunctions. To alleviate this problem, we introduce PosCap, a part-of-speech (POS) assisted video captioning model. Leveraging POS information as the prior knowledge, PosCap enhances word prediction by focusing on distinctive multimodal features for different word generation. Specifically, we introduce a POS prediction module to predict the POS of the next word. The predicted POS information guides the attention mechanism to better integrate different modalities information, therefore effectively improving word generation. Experimental results on two benchmark datasets, convincingly demonstrate that PosCap outperforms existing methods in generating coherent and grammatically correct video descriptions.