Do We Still Need Text Features for Video Retrieval in the Era of Vision-Language Models?
摘要
In video retrieval, text features have been considered essential conventionally, primarily in the form of video metadata, captions, and Automatic Speech Recognition (ASR) outputs, etc. However, with the rapid progress on multimodal large language models that have strong capability on understanding video and audio, do we still need text features for video retrieval when using state-of-the-art vision-language models as the backbone for dense retrieval? We approach the question based on MultiVENT 2.0, a video retrieval dataset that provides text, audio, and video information per document, and train a series of multimodal embedding models based on OmniEmbed. Comparing models trained and evaluated on different input modalities, we find that models based on multimodal inputs alone (video and audio) achieve strong effectiveness, showing great potential for end-to-end optimization in video retrieval without relying on additional automatically-extracted or human-written text features.