Video retrieval is a crucial problem in computer vision and natural language processing, aiming to search and retrieve videos (or scenes, shots, and frames) based on a user given a video query. Among recent approaches for this task, keyframes-based content retrieval with a vision language model has shown significant efficiency and potential results. In this study, we evaluate various recent vision-language models for their effectiveness in replacing traditional CLIP in video retrieval problems with textual descriptions. We also introduced a customized keyframe extraction module using visual similarity to solve this issue. The findings indicate that, in terms of retrieval performance, EVA-CLIP and SigLIP notably surpass CLIP and alternative models when evaluated on the large-scale video datasets V3C. Furthermore, we introduce a customization keyframe extracting module to address the issue of information lacking in the default keyframe given in the dataset when compared to textual queries. Our customized keyframe extraction not only decreased the number of keyframes but also significantly improved retrieval outcomes compared to the default keyframes provided in the V3C dataset and extracted by Cineast.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Video Retrieval System with EVA-CLIP and Customized Keyframes Extraction

  • Khanh-An C. Quan,
  • Qui Ngoc Nguyen

摘要

Video retrieval is a crucial problem in computer vision and natural language processing, aiming to search and retrieve videos (or scenes, shots, and frames) based on a user given a video query. Among recent approaches for this task, keyframes-based content retrieval with a vision language model has shown significant efficiency and potential results. In this study, we evaluate various recent vision-language models for their effectiveness in replacing traditional CLIP in video retrieval problems with textual descriptions. We also introduced a customized keyframe extraction module using visual similarity to solve this issue. The findings indicate that, in terms of retrieval performance, EVA-CLIP and SigLIP notably surpass CLIP and alternative models when evaluated on the large-scale video datasets V3C. Furthermore, we introduce a customization keyframe extracting module to address the issue of information lacking in the default keyframe given in the dataset when compared to textual queries. Our customized keyframe extraction not only decreased the number of keyframes but also significantly improved retrieval outcomes compared to the default keyframes provided in the V3C dataset and extracted by Cineast.