Diseases in plants are common with changing environmental conditions. These diseases infect various parts of the plant resulting in long-term economic losses if left unnoticed. The plant’s leaf easily responds to diseases as various demarcations appear on them. Leaf examination is the most traditional way to detect diseases, with the limitation of being extremely labor and time intensive. This study presents an approach for classifying apple leaf diseases using a deep learning-based pipeline. Healthy, Rust, Scab and Multiple disease are the four classes that are used in this study. The proposed deep CNN architecture categorizes the input image into one of the four classes. Several ensemble approaches are presented here, comparing the performance of each on the multiple disease class. This work explores the use of different training methods involving freezing and unfreezing. The proposed architecture based on CoAtNet achieves an overall accuracy of 95% and an accuracy of 68% on the multiple disease class on the publicly available plant pathology 2020 dataset. It appears that the proposed model has the potential for the real-time diagnosis of apple leaf diseases significantly.