Developing an efficient video event retrieval system is crucial due to the rapid growth of video-sharing platforms, resulting in a vast increase in the number and volume of video content. Traditional techniques used in previous VBS competitions, such as Optical Character Recognition (OCR) and Object Detection, primarily relied on metadata. However, current state-of-the-art methods utilize pre-trained models like CLIP and BLIP. These models effectively link textual queries with image data, but user queries are sometimes ambiguous, leading to possible mismatches between user expectations and the retrieved content. Moreover, many duplicate frames may not contribute new information when extracting keyframes, increasing search times and storage costs. To overcome these challenges, we present AViSearch, an advanced video event retrieval system developed for AI Challenge HCMC 2024. AViSearch leverages Gemini to optimize queries and generate more diverse queries from the user’s original input, facilitating a more detailed need. We also optimize keyframes, reducing storage requirements and improving the efficiency of the search process. Additionally, our system incorporates various retrieval techniques, including Optical Character Recognition (OCR), Automatic Speech Recognition (ASR), object detection, and image similarity, providing a comprehensive toolkit for achieving more accurate and efficient search results.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

AViSearch: A Multimodal Video Event Retrieval System via Query Enhancement and Optimized Keyframes

  • Nguyen Huu Hoang Long,
  • Tran Thi Cam Giang,
  • Tran Nhu Cam Nguyen,
  • Phan Hoang Phuoc,
  • Dang Huu Phat,
  • Tien-Huy Nguyen

摘要

Developing an efficient video event retrieval system is crucial due to the rapid growth of video-sharing platforms, resulting in a vast increase in the number and volume of video content. Traditional techniques used in previous VBS competitions, such as Optical Character Recognition (OCR) and Object Detection, primarily relied on metadata. However, current state-of-the-art methods utilize pre-trained models like CLIP and BLIP. These models effectively link textual queries with image data, but user queries are sometimes ambiguous, leading to possible mismatches between user expectations and the retrieved content. Moreover, many duplicate frames may not contribute new information when extracting keyframes, increasing search times and storage costs. To overcome these challenges, we present AViSearch, an advanced video event retrieval system developed for AI Challenge HCMC 2024. AViSearch leverages Gemini to optimize queries and generate more diverse queries from the user’s original input, facilitating a more detailed need. We also optimize keyframes, reducing storage requirements and improving the efficiency of the search process. Additionally, our system incorporates various retrieval techniques, including Optical Character Recognition (OCR), Automatic Speech Recognition (ASR), object detection, and image similarity, providing a comprehensive toolkit for achieving more accurate and efficient search results.