Multimodal Video Understanding and Temporal Grounding with Large Language Model
摘要
In the era of exponential growth in unstructured multimedia data, effectively understanding and retrieving relevant information from videos remains a significant challenge. This project proposes a novel framework that leverages the power of large language models (LLMs) to achieve multimodal video understanding and context retrieval. The framework comprises three key modules: a visual module for object detection using a fine-tuned YOLOv8 model, an audio module for transcription and speaker diarization via OpenAI's Whisper fine-tuned on ULCA ASR Corpus and iCSTD IIIT-H ASR Corpus for Telugu language, and a language model module that comprehends the multimodal data and answers user queries. At the core of this framework lies the Mistral Uncensored Fine-Tuned LLM, which is equipped with a context corpus to comprehend the video's visual and audio components. The LLM's capabilities are further enhanced by incorporating timestamp retrieval, allowing for direct time-based responses to user queries. The proposed system is designed to be versatile, capable of understanding various video genres and enabling users to search and ask questions about the provided video content. Through this multimodal approach, the framework aims to bridge the gap between unstructured video data and human understanding, facilitating efficient information retrieval and fostering a seamless user experience. By combining state-of-the-art computer vision, speech recognition, and language modeling techniques, this project paves the way for more effective video comprehension and query answering systems.