MAVERICS: Multimodal Advanced Visual Event Retrieval with Integrated CPU-Optimized Search
摘要
The increasing volume of visual data in news archives and media sources poses significant challenges for efficient event retrieval. This paper presents a multimodal approach to tackle the problem of Event Retrieval from Visual Data. Our system integrates several techniques to process diverse query types, including text, image, and video. For image-text retrieval, the BLIP2 model is used to embed both images and text descriptions. In cases where queries are in Vietnamese, we employ the pre-trained VietAI/envit5-translation model to translate prompts into English before processing them with BLIP2. Object detection is handled by YOLOWorldv2, and text extraction from images utilizes PP-OCRv3 and VGG Transformer. Additionally, WhisperX is employed for audio-to-text conversion. Embeddings from textual data, whether derived from OCR or audio, are generated using sentence-transformers/all-MiniLM-L6-v2. These embeddings are indexed using Usearch, enabling fast and efficient retrieval. Furthermore, we developed a high-speed temporal search mechanism that calculates scores and combinations for consecutive related frames to improve performance in temporal queries. The system is capable of running efficiently on CPUs, with a maximum query processing time of 2 s for advanced queries, such as Temporal search, which require multiple models to run consecutively, making it a scalable solution for large-scale video data retrieval. Additionally, we have built a user-friendly interface using Streamlit, enabling users to easily interact with and utilize the system.