Banana ripeness classification influences the food supply chain from production to marketing; thus, one has to undergo classification methods that are subjective and inconsistent in their application. This study proposes a deep learning-based system that uses MobileNetV2 architecture to automatically detect banana ripeness in four stages. The combined dataset of 3,600 real and synthetically constructed pictures was class balanced; all pictures were of high resolution, i.e., 224 × 224 pixels. To make the model more generalized under varying environmental settings, data augmentation was integrated, thus resulting in some 12,000 training samples. For categorical classification, label encoding was done via one-hot encoding, and weighted loss functions were used to avoid class imbalance. Transfer learning and fine-tuning were employed using ImageNet pre-trained weights. This model, using the Adam optimizer with categorical loss, was able to attain 92% accuracy on 699 testing samples, along with macro-averaged precision, recall, and F1 scores of 0.90, 0.91, and 0.91, respectively. The gradual unfreezing of the final layers towards the completion of the training further adds to the performance. A relatively lightweight computational profile of 0.56 GFLOPS and an inference time of 8.20 ms per sample make the proposed solution an ideal candidate for real-time deployment on mobile and edge devices. Compared to existing CIDIS CNN, our model achieved higher accuracy with fewer parameters and greater efficiency, demonstrating a superior balance between performance and computational cost.