AViSearch: A Multimodal Video Event Retrieval System via Query Enhancement and Optimized Keyframes
摘要
Developing an efficient video event retrieval system is crucial due to the rapid growth of video-sharing platforms, resulting in a vast increase in the number and volume of video content. Traditional techniques used in previous VBS competitions, such as Optical Character Recognition (OCR) and Object Detection, primarily relied on metadata. However, current state-of-the-art methods utilize pre-trained models like CLIP and BLIP. These models effectively link textual queries with image data, but user queries are sometimes ambiguous, leading to possible mismatches between user expectations and the retrieved content. Moreover, many duplicate frames may not contribute new information when extracting keyframes, increasing search times and storage costs. To overcome these challenges, we present AViSearch, an advanced video event retrieval system developed for AI Challenge HCMC 2024. AViSearch leverages Gemini to optimize queries and generate more diverse queries from the user’s original input, facilitating a more detailed need. We also optimize keyframes, reducing storage requirements and improving the efficiency of the search process. Additionally, our system incorporates various retrieval techniques, including Optical Character Recognition (OCR), Automatic Speech Recognition (ASR), object detection, and image similarity, providing a comprehensive toolkit for achieving more accurate and efficient search results.