It is critical to identify and then classify diseases in pomegranate fruits to ensure a high-quality yield and reduce agricultural losses. Regretfully, large-scale monitoring is difficult since traditional methods of identifying illnesses are time-consuming and often require specialist knowledge. Advanced methods for automated disease classification have been made possible in recent years by Machine Learning (ML) and Deep Learning (DL). Transfer learning (TL), one of these approaches, has proven to be successful in addressing particular agricultural issues with sparse data by utilizing pre-trained models on big datasets. Here a method for classifying pomegranate fruit problems called Hybrid Transfer Learning (HTL) is proposed. This approach integrates domain-specific feature extraction with the benefits of TL and fine-tuning approaches. Images of both healthy and sick pomegranate fruit are fed into pre-trained Convolutional Neural Networks (CNNs), such as Res-Net, Inception, SVM, K-nearest neighbor to extract generalized visual information. Following this, a process of fine-tuning combines these features with a specially designed disease categorization module that is tailored to the distinct qualities of pomegranate disorders. Affected crops have caused the agricultural field to suffer and also affect the economy. However, the reason behind the infected crops may be crop diseases. To resolve this issue, pomegranate fruit disease classification based on hybrid transfer learning is chosen. Different machine learning algorithms such as random forest classifier and K-nearest neighbor algorithm with custom deep learning algorithms are used. Our run-time training accuracy is 97.5%.