Accurate fruit segmentation from complex backgrounds, shadows, and overlapping objects, mainly leaves, can degrade the effectiveness of overall classification. Overcoming those obstacles can aid in automated fruit detection, efficient fruit disease identification, and decreased fruit rotten issues. To solve these fundamental issues, this study presents a hybrid framework over a comprehensive set of 30 fruits assessed using a publicly available dataset. This framework uses UNet as an image segmentation model to determine the region of interest (ROI) by generating masks of input images from a complicated background. This assists in the modified MobileNet-v2 to improve fruit categorization. Additionally, YOLO v8 is used to detect fruits from a group of fruits. Further understanding of prototypical behavior is achieved through the use of Gradient-weighted Class Activation Mapping (GRAD-CAM), which graphically identifies the crucial areas of the input images that influence the model’s judgments. The mobileNet-v2 yields a classification accuracy of 96. 45%, followed by a precision, recall, and F1 score of 0.97, 0.96, and 0.96, respectively. The segmentation IoU and dice scores are 95.69% and 97.89%, respectively. In object detection, YOLO v8 gave an average mAp score of 0.95. This hybrid approach can automate procedures such as fruit quality assessment and categorising fruits for fast transportation, and its transparency in classification could be useful in AI-powered agricultural applications.