Deep neural networks (DNNs) have become powerful computational models that consistently outperform traditional machine learning approaches in analyzing both structured and unstructured data. In particular, convolutional neural networks (CNNs) have been proven successful for image classification by automatically learning hierarchical representations, ranging from low-level edges and textures to high-level semantic features. However, training CNNs from scratch is often infeasible due to the lack of annotated data, coupled with the heavy computational burden, problems that are particularly acute in specialized domains such as medical imaging, remote sensing, or industrial defect detection. To avoid these problems, researchers utilize pre-trained CNN architectures, such as VGG, ResNet, Inception, and DenseNet, which are trained on large-scale datasets like ImageNet. These models can be generalized, and transfer learning can be used to transfer the rich feature representations to new tasks, saving both training cost and training time. However, domain mismatch remains a significant challenge, as pre-trained models trained on natural images may not generalize directly to highly domain-specific datasets. To solve this, we develop a hybrid framework that incorporates transfer learning with Harris Hawks Optimization (HHO). While pre-trained CNNs can transfer to a target domain efficiently using transfer learning, HHO is a strong optimizer for fine-tuning the key network parameters and hyperparameters. This approach not only increases the convergence rate of training but also the classification accuracy and generalization of the model. Experimental results validated that the joint of HHO and transfer learning (TL-HHO-CNN) significantly outperformed models trained from scratch and even surpassed the standard fine-tuning methods, offering better robustness, lower training time, and excellent performance in domain-specific tasks.