In the agricultural sector, operations like harvest monitoring and crop yield estimation depend heavily on the identification and classification of fruit and vegetable growth phases. It helps the farmers to adopt appropriate measures on time to improve the crop yield. Our study proposes a deep learning-based approach for classifying pomegranates into five distinct stages: bud, flower, early-fruit, mid-growth, and mature. We have employed two versions of the YOLO model (v5 and v8) to detect and classify the pomegranates. The dataset that we used to train and test the models included 4685 images showing pomegranates at various phases of growth (Bud, Flower, Early-Fruit, Mid-Growth, and Mature). We have used mAP, precision, and recall to evaluate our models. Our results show that v8 exhibited a 2.6% improvement in mAP (with an overlap of 0.5–0.95) score compared to v5. Both models (v5 and v8) are able to identify the mature fruits very accurately and bud is the class with the least mAP (with an overlap of 0.5) and mAP (with an overlap of 0.5–0.95) scores.