CLIP-Enhanced Lifelog Retrieval System: Robust Multi-modal Media Search with Real-Time Performance
摘要
In this paper, we introduce a robust media retrieval system designed to address the challenges posed by large-scale, multi-modal data retrieval tasks, particularly in the context of image retrieval, which plays a crucial role in surfacing key moments from vast media datasets. Efficient and accurate image retrieval is essential for individuals to navigating and retrieving relevant events, making it a vital component of any advanced retrieval system. Our system builds on retrieval models by integrating advanced features, more specific like CLIP-based image search, YOLOv8 for precise object detection, and temporal search to handle long and complex queries. Key optimizations include enhanced visual similarity search and an intuitive, interactive interface that ensures fast and efficient query results. By storing extracted features in the Milvus vector database, the system achieves significant speed improvements in retrieval, enabling real-time performance. Benchmarking our model at the AI Challenge (AIC24) in Ho Chi Minh City, the system demonstrated top-tier results, particularly excelling in KIS-type queries, where it achieved 100% frame retrieval accuracy (15/15) with an average query response time under 30 s. These results highlight the effectiveness of our system in handling diverse and complex queries, making it a valuable tool for lifelog retrieval tasks, especially in improving the user experience for both novice and expert users. Our code is publicly available at https://github.com/trnKhanh/AIC24 .