<p>As the core reference for pavement distress positioning, the detection and recognition accuracy of road milestones directly determines the accuracy of distress localization. In inspection images, road milestones often account for less than 1% of pixels and are interfered with by factors like low resolution and complex backgrounds, making traditional algorithms struggle to balance detection precision and real-time performance, thus posing severe challenges to real-time detection and text recognition of road milestones. To address the difficulties in small-target milestone recognition and text extraction, this paper proposes a lightweight algorithm RMTD-YOLO (Road Milestone and Text Detection—You Only Look Once) integrated with four improved modules, which achieves model lightweighting while effectively enhancing real-time detection accuracy, efficiency, and environmental adaptability of milestones. A dataset of 1228 highway milestone images with varying resolutions and environments is constructed to improve the model’s detection accuracy and generalization. Horizontal comparison experiments with mainstream algorithms and ablation experiments on the four modules verify the model’s effectiveness. Results show RMTD-YOLO achieves 0.9629 precision, 0.9268 recall, 0.9445 mAP50, and 0.9537 mAP50:95, which are 6.14%, 11.38%, 8.96%, and 4.54% higher than YOLOv8n, respectively, with 234.93 FPS and a 15% smaller size, achieving a better balance between precision and real-time performance. Integrated with PaddleOCR, it forms an integrated “real-time detection-text recognition” framework for intelligent milestone information extraction, meeting deployment requirements of resource-constrained scenarios like vehicle-mounted real-time inspection.</p>