The exponential growth of video data has created significant challenges in retrieving relevant information from long-form multimedia content. Existing systems focus on short-form videos and rely on semantic vector search or shallow text matching, which struggle to capture contextual relationships in the video data. This study proposes a hybrid, context-aware retrieval framework that integrates neural text embeddings with knowledge graph based reasoning to improve accuracy and semantic relevance in video segment search based on a natural language user queries. The system performs contrastive segmentation and extracts entities and relations from video segments using the Relation Extraction By End-to-end Language generation (REBEL) model to store them in Neo4j. Retrieval combines graph traversal and vector similarity through a weighted fusion model, enabling both structural and semantic alignment with the user query. Experimental results demonstrate that hybrid retrieval significantly outperforms standalone semantic or graph-based methods, achieving the highest MRR of 0.84. Recall@K analysis further highlights the system’s ability to retrieve relevant content, with improvements observed as K increases. Overall, the framework offers an effective and scalable solution for multi-modal context-driven video search based on user query, with an opportunity for enhanced modality and LLM future extensions.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Context-Aware Video Segment Search: Integrating Knowledge Graphs for Query-Based Retrieval

  • Rimsha Fatima,
  • Khizar Mohamed Zubair Sait,
  • Goutham Kolla,
  • Satya Kiranmai Tadepalli

摘要

The exponential growth of video data has created significant challenges in retrieving relevant information from long-form multimedia content. Existing systems focus on short-form videos and rely on semantic vector search or shallow text matching, which struggle to capture contextual relationships in the video data. This study proposes a hybrid, context-aware retrieval framework that integrates neural text embeddings with knowledge graph based reasoning to improve accuracy and semantic relevance in video segment search based on a natural language user queries. The system performs contrastive segmentation and extracts entities and relations from video segments using the Relation Extraction By End-to-end Language generation (REBEL) model to store them in Neo4j. Retrieval combines graph traversal and vector similarity through a weighted fusion model, enabling both structural and semantic alignment with the user query. Experimental results demonstrate that hybrid retrieval significantly outperforms standalone semantic or graph-based methods, achieving the highest MRR of 0.84. Recall@K analysis further highlights the system’s ability to retrieve relevant content, with improvements observed as K increases. Overall, the framework offers an effective and scalable solution for multi-modal context-driven video search based on user query, with an opportunity for enhanced modality and LLM future extensions.