Road defect detection is considered a prerequisite for maintaining road quality and a necessary measure to ensure driving safety. In addressing the challenges posed by existing methods’ inefficiency, inaccuracy, and high cost, an improved road defect detection algorithm grounded in the YOLOv8 framework is proposed, denoted as YOLOv8n-CSW. First, we integrate the CBAM attention mechanism into the backbone network’s C2f module, now termed the C2fCBAM module, to enhance feature extraction capabilities and suppress irrelevant information interference. Then the lightweight convolutional method GSConv and cross-level partial network module, VoV-GSCSP, can be flexibly integrated as a sleek and efficient slim-neck to seamlessly displace the initial neck network, thus optimizing the overall architecture and performance, ensuring accuracy while reducing computational complexity. Moreover, substituting the loss function with Wise-IoU enables focusing on the anchor frame of common quality and improving overall detector performance. The experimental results conclusively present that on the RDD2022 dataset, the mAP50 of the enhanced algorithm achieves 56.9%, surpassing the original algorithm accuracy by 1.7%. The computational volume is reduced to 7.3G, representing a 9.8% decrease. Additionally, the algorithm achieves an impressive frame detection rate of 83 frames per second (FPS) enabling accurate detection of road defects in real-time.