Book genres are not always clearly defined, and classifying them based solely on visual or textual patterns can be unreliable. While recent models attempt to improve genre classification by combining both cues, key limitations remain in the design of input representations and model architectures. Motivated by these challenges, this paper proposes CoVirNet, a multimodal classification system that predicts a book’s genre using its cover image and title. The model processes two color space representations of the image through a hybrid architecture, in which a Convolutional Neural Network (CNN) and a Vision Transformer (ViT) operate in parallel to extract both local and global visual features. Concurrently, the book title is processed using a BERT-based encoder, which captures deep semantic signals from the text. Evaluated on the BookCover30 dataset, CoVirNet outperforms state-of-the-art models and its baseline variants, achieving 65.23% Top-1 accuracy and 84.70% Top-3 accuracy. These results highlight the benefits of color space fusion, architectural hybridization, and deep text modeling in improving multimodal book genre classification.