Two-Pass Architecture for Video Processing in LLMs
摘要
While Large Language Models have become increasingly adept at trying to parse multimodal content, they still struggle to parse video content. Generally, the root cause of this issue is the sheer size of videos, especially since each video has up to 60 frames per second. A general solution that is implemented to parse videos is to fine-tune custom large visual language models specifically for videos. However, although these approaches result in robust and efficient models, due to the lack of extensive video-annotated datasets and intensive training requirements, they aren’t quite scalable. We propose a custom framework for using pre-trained LVLMs that are already fine-tuned for images within a chained two-iteration architecture that parses only selective keyframes that are extracted from the video. While current annotated video training-based models are easily bottlenecked in terms of dataset availability, our proposed system, being an extension of pre-trained image-based LVLMs, is modular and isn’t dependent on custom fine-tuning, overcoming the issue of limited datasets given the presence of existing high-quality image datasets. Also, since only keyframes (up to 3) are parsed within the system, the total processing computational effort and time are highly reduced as compared to general video processing systems since only selective data are processed. In this paper, we introduce a new architecture called a two-pass architecture, which is recurrently chained across all frames for processing and inter-frame context chaining to parse videos.