Potatoes are one of the most cultivated and consumed crops worldwide. Disease control in potato plants is crucial for efficient production of potatoes. In this paper, we propose a method to detect and classify diseases from the images of potato leaves. For this, we propose a deep neural architecture to classify the plant diseases from leaf images. We have used the PSOK image segmentation method to segment the leaf images into meaningful features. Data augmentation techniques like flipping, rotating, skewing, and zooming are used to increase the dataset. We compare the performance of our proposed model with pretrained model. We show that our proposed architecture combined with XGBoost trained on the augmented dataset achieves an accuracy of 97.81%, outperforming the existing methods.