Dental cavity detection is one of the major problem in healthcare sector, which greatly affects oral health and general well-being. Early and precise detection, further progression of decay can be addressed, with less invasive treatment and better results for the patients. This paper discusses the effectiveness of two latest deep learning architectures, You Only Look Once v8 (YOLOv8) and YOLO11, for real-time object detection and instance segmentation over a dataset of 2495 images, formatted in Common Objects in Context (COCO) using Roboflow. The dataset is divided into training(1991), validation (254), and testing (250) sets. The performance is assessed using metrics such as mean Average Precision (mAP), precision and recall. For instance segmentation, YOLOv8 achieved a precision of 52.4%, recall of 53.6%, mAP@50 of 50.5%, and mAP@50–95 of 34.1%, while YOLO11 showed higher precision (62.2%) but lower recall (46.0%), with mAP@50 at 48.2% and mAP@50-95 at 33.8%. For object detection, YOLOv8 outperformed YOLO11 in both precision (53.7% vs. 42.9%) and mAP@50 (46.8% vs. 42.9%), though YOLO11 had slightly higher recall (48.5%). The findings show that both models can effectively detect dental cavity, with YOLOv8 showing superior overall performance. The study demonstrates the importance of dataset customization and model optimization to ensure more reliable detection and segmentation.