Interactive Video Retrieval System for AI Challenge 2024 Using CLIP, RAM++, and LLM-Enhanced Tag Matching
摘要
In this paper, we present an interactive video retrieval system developed for the AI Challenge 2024. The system offers multi-modal search functionality, allowing users to search using text, images, or tags. At its core, the system leverages CLIP (Contrastive Language-Image Pre-training) to enable efficient video retrieval from both natural language and image-based inputs. For tag-based queries, we incorporate the state-of-the-art image tagging Recognize Anything Plus (RAM++) model. However, due to the large number of tags produced by RAM++, it becomes impractical to manually select the most relevant tag of the input query. To address this challenge, we use the Gemini Large Language Model (LLM) to automatically select the most appropriate tags for the given query. Additionally, we describe our temporal search algorithm, which further enhances retrieval performance. Our experiments show that this combination of models provides a scalable and high-performance solution for video search applications in real-world scenarios.