Dental cavity identification using advanced image processing and machine learning techniques, especially through X-rays, plays a crucial role in early diagnosis and treatment planning. Traditional detection systems often suffer from high error rates and low accuracy. To address these challenges, a sophisticated model based on Riemannian Residual Neural Networks with Improved Sooty Tern Optimization (RR2Net-ImSTOpt) is proposed. The model uses the DENTEX dataset for analysis, incorporating noise reduction and image enhancement using Guided Box Filtering (GBF). Feature extraction is performed using the Inception Vis-Transformer, followed by optimization of RR2Net’s weight parameters via the Improved Sooty Tern Optimization Algorithm. This approach achieves impressive results with a recall of 99.8% and an accuracy rate of 99.9%, surpassing current methods in accuracy and reducing false positives. RR2Net-ImSTOpt’s capability to handle large medical datasets makes it an ideal solution for clinical dental cavity detection, enhancing diagnostic efficiency and precision.