Hydroponic lettuce is known for its rich in calcium, vitamin A and iron. Identification of good leaves is a challenge for supermarkets. An automatic process is needed to classify these leaves. In this paper, the four classes being considered are Fully Nutritious, Nitrogen (N), Potassium (K) and Phosphorous (P) Deficient leaves. Convolutional Neural Networks (CNN) can capture image patterns. This feature is explored to recognize the four classes. CNN is implemented for various numbers of layers (2, 3, and 4) to determine the impact of the number of layers on classification accuracy. Further, transfer learning is also explored for classification. In transfer learning, the first model is trained with a well-established data set. This is used as the starting point for the new task. This is useful when the data set is small. Transfer learning is useful to improve the accuracy of the classification. The study highlights the importance of automated sorting systems in hydroponic lettuce production and the potential of machine learning and deep learning algorithms in improving the accuracy and efficiency of the classification process. The performance of all the methods is assessed using accuracy, precision, recall and F1 score. Experimental results indicate that the proposed method gives better accuracy for four classes.