Query-Driven Video Summarization via Shot-Level Caption Generation
摘要
Query-driven video summarization creates concise video summaries tailored to the user’s textual queries. Unlike traditional summarization methods that produce generic video summaries, query-driven approaches focus on the relevance of the summarized content to the user’s specific interests. These methods leverage Natural Language Processing and Computer Vision techniques to analyze and understand both the video and the query, enabling the extraction of keyshots relevant to the user’s interest. This research introduces a methodology for query-driven video summarization via shot-level caption generation. The method involves segmenting the input video and generating shot-level captions using a pretrained Bootstrapped Language-Image Pre-training (BLIP) model. Both the generated captions and the user-inputted textual queries are encoded into token-level features using a Contrastive Language-Image Pre-Training (CLIP) text encoder. Contextual mapping is performed to compare the captions with the input queries, and matching keyshots are extracted. The performance of the proposed method is evaluated using F1-scores. The qualitative and quantitative experimental results demonstrate that the proposed technique successfully produces query-driven video summaries.