This chapter proposes a computer vision-based technique for defect detection in Mango fruits by utilizing the efficiency of deep learning-based object detection models. A dataset consisting of 1600+ samples with defects at various stages of ripening is considered for experimentation. The dataset is collected with the help of a specialized controlled environment adopted in industries during the production process for grading and sorting. Additionally, two publicly accessible datasets, which comprise images of fruits with defects captured in real-world scenarios, are also considered for experimentation so as to demonstrate the robustness of the proposed object detection model. A customized deep learning-based object detection model is devised to perform defect detection. The model’s initial layers perform the data preprocessing and augmentation followed by object detection and, finally, classification of detected regions into classes—defects, bruises, rotten ripe, ripe, partially ripe, and ripe stages. The experimentations on the proposed and publicly available datasets achieve a mean Average Precision (mAP) of 64.8%, precision of 76.3%, and recall of 65.01%, respectively. An object loss of less than 2.5% is observed with a class loss of less than1.5% and box loss of 1.7% for 300 epochs at a learning rate of 0.01. The visual results of experiments are prone to be consistent with the subjective assessment results.