News Video Captioning with Asynchronous Visual and Audio Information
摘要
In news video captioning, visual and audio modalities are often asynchronous along the timeline. Audio outside the bounds of the described video segment frequently contains essential contextual information, including people, locations, or the overall topic of the news, which are crucial for generating accurate video captions. In this paper, we analyze the news video captioning dataset News-11k by comparing manual annotations derived from different modalities, and demonstrate that asynchronous audio information provides significant information for video caption generation. Motivated by this observation, we propose a framework that leverages off-the-shelf multimodal large language models to integrate asynchronous visual information and audio information. Specifically, asynchronous audio is first transcribed by an automatic speech recognition model. The audio information is then summarized and fused with visual content using the multimodal model to generate video captions, integrating both visual and audio information. Experimental results show that, compared with using only synchronous visual-audio information, incorporating extended asynchronous audio improves the quality and completeness of generated news video captions.