A Spatio-Temporal Explainable Deep Learning Approach for Frame Classification from Book Flipping Videos
摘要
Book flipping videos present a challenging domain for information extraction due to the dynamic nature of page turns and the need to identify frames with clear text visibility. This research proposes a novel approach for frame classification in book flipping videos, combining the power of 3D Convolutional Neural Networks (3D CNNs) with eXplainable AI (XAI) techniques. Specifically, the mVGG network, a pre-trained 3D CNN architecture, is employed to capture spatio-temporal features and classify frames as “good” or “bad” based on the presence of page turns and text visibility. Traditional frame classification methods often fail to capture the crucial temporal context of page turns. Our proposed approach addresses this limitation by processing entire video sequences as 3D volumes, enabling the mVGG network to learn complex spatio-temporal features that are indicative of page turns and text visibility. To enhance the transparency and interpretability of the model's predictions, we integrate LIME (Local Interpretable Model-agnostic Explanations), a prominent XAI technique. LIME allows us to generate local explanations for individual frame classifications, highlighting the specific regions of the video that contributed most to the model's decision. This provides valuable insights into the model's reasoning process and aids in understanding its strengths and limitations. This research offers significant benefits for tasks like automated book digitization, digital library creation, and educational technology applications. By accurately classifying frames and providing interpretable explanations for the classifications, our approach can significantly improve the efficiency and reliability of extracting information from book flipping videos.