Road surfaces are prone to cracks and potholes due to environmental variations and material aging. Existing object detection algorithms exhibit inadequate adaptability to morphological diversity, feature ambiguity, and environmental interference in road damage, coupled with a persistent reliance on post-processing steps. To address these challenges, we propose an end-to-end real-time road damage detection algorithm named RDD-DETR. To address feature ambiguity and background interference, an image enhancement method named M-USM and a residual context enhancement module (RCE Block) are designed to strengthen feature extraction capabilities in complex scenarios. A deformable attention-based feature interaction (DAIFI) module is constructed to optimize high-level feature interaction, while a cross-guided attention fusion (CGAF) module is established to achieve adaptive multi-scale fusion through bidirectional feature complementarity. The end-to-end architecture is implemented to eliminate the dependency on NMS post-processing inherent in existing YOLO-series algorithms. Experiments demonstrate that RDD-DETR achieves a 5.2% improvement in accuracy over baseline while maintaining speed comparable to YOLO-series algorithms.