Public infrastructures, including buildings, bridges, and roads, are critical to urban development. Structural inspection is a crucial process that need to be conducted regularly to maintain safety and prevent deterioration. Traditional manual inspection methods are time-consuming, prone to errors, and inefficient. This research introduces an automated framework for detecting and analyzing concrete defects using a transformer-based model to address these limitations. There are four main contributions: (i) a transformer-based detection system that utilizes the Deformable DETR model to improve the detection performance, (ii) the collection of a large-scale dataset containing four common concrete defect types, (iii) the integration of various components into the original architecture to boost detection accuracy, and (iv) a method to visualize the deformable attention to interpret the model’s predictions. The experimental results highlight that the proposed model outperforms state-of-the-art object detection methods with a mean Average Precision (mAP) of 0.63. Therefore, it significantly improves automated concrete defect inspection and can be applied to real-life inspection applications.