The degradation of road infrastructure is a worrying global issue, meaning that timely regular maintenance is needed to ensure that it stays safe and the economy sustainable. However, manually annotating road damage datasets is cumbersome and slow, creating a bottleneck for the development of automated detection systems. In this work, we proposed a self-supervised learning-based approach that uses limited annotated data for training. This approach consists of two distinct periods: the pretext task and the downstream task. In the pretext task, the model learns robust image representations by training on an unlabeled dataset of road damage images. A pre-trained model is finetuned with a relatively small amount of labelled images for road damage classification. The results show that the proposed method outperforms prior art for road damage classification. The work leads the way to scalable and economical solutions for the maintenance of roads and shows how AI systems can change monitoring infrastructure. The proposed self-supervised learning-based approach achieves 99.26% accuracy on the Major South African Highway Binary Crack Dataset, surpassing the previous 98%, with precision, recall, and F1-score above 99% and a Cohen’s Kappa score of 98.51%, demonstrating state-of-the-art performance in road damage detection.