Investigating Neural Networks and Transformer Models for Enhanced Comic Decoding

Comic books, merging art with narrative, continue to captivate readers, cinema producers, and collectors, maintaining their allure as a cherished form of visual storytelling across decades. Comic image segmentation is a pivotal aspect in the digital transformation of comics. Leveraging heuristic approaches, neural network-based model (YOLO), and innovative transformer-based architectures (GroundingDINO, SAM), our research aims to autonomously segment comic pages into fundamental components: panels, comic characters, and text areas. To this end, we further trained YOLOv5 and YOLOv8 models to identify these components, while transformer-based models employed prompts to retrieve them. By comparing their outputs across three well-known datasets (eBDtheque, DCM772, Manga109) and using different metrics (Precision, Recall, Average Precision), we conclude that pre-trained self-supervised transformer models can competently outperform state of the art approaches, which often require further fine-tuning to achieve comparable results.