In video retrieval, text features have been considered essential conventionally, primarily in the form of video metadata, captions, and Automatic Speech Recognition (ASR) outputs, etc. However, with the rapid progress on multimodal large language models that have strong capability on understanding video and audio, do we still need text features for video retrieval when using state-of-the-art vision-language models as the backbone for dense retrieval? We approach the question based on MultiVENT 2.0, a video retrieval dataset that provides text, audio, and video information per document, and train a series of multimodal embedding models based on OmniEmbed. Comparing models trained and evaluated on different input modalities, we find that models based on multimodal inputs alone (video and audio) achieve strong effectiveness, showing great potential for end-to-end optimization in video retrieval without relying on additional automatically-extracted or human-written text features.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Do We Still Need Text Features for Video Retrieval in the Era of Vision-Language Models?

  • Jiaqi Samantha Zhan,
  • Crystina Zhang,
  • Shengyao Zhuang,
  • Xueguang Ma,
  • Jimmy Lin

摘要

In video retrieval, text features have been considered essential conventionally, primarily in the form of video metadata, captions, and Automatic Speech Recognition (ASR) outputs, etc. However, with the rapid progress on multimodal large language models that have strong capability on understanding video and audio, do we still need text features for video retrieval when using state-of-the-art vision-language models as the backbone for dense retrieval? We approach the question based on MultiVENT 2.0, a video retrieval dataset that provides text, audio, and video information per document, and train a series of multimodal embedding models based on OmniEmbed. Comparing models trained and evaluated on different input modalities, we find that models based on multimodal inputs alone (video and audio) achieve strong effectiveness, showing great potential for end-to-end optimization in video retrieval without relying on additional automatically-extracted or human-written text features.