Road damage can greatly affect road security. Due to the development of deep learning, a number of researchers have applied this in road defect detection research. However, due to the specific characteristics of road images small objects, complex background scenes, and large variances in object size the task of road defect detection still faces great challenges after advances in object detection. Detecting and recognizing these road defect objects is very difficult because of their relatively small sizes, low contrast, and poor quality feature images. To solve such problems, a novel road defect detection network based on polarity-aware linear attention and multi-cognitive visual adapter is proposed. Firstly, we present a polarity-aware linear attention and an enhanced Pomo-Attention module to tackle the small-size-object problem in road defect cases in order to improve the detection performance. Second, we present the DyMon module, which employs a modified Multi-Cognitive Visual Adapter for better parameter fine-tuning. Finally, we redesign the multiscale SaE attention module to enhance the detection capability for objects of different sizes. Extensive experiments on benchmark datasets demonstrate that the proposed method achieves a highest mAP@50:95 (%) score of 44.13%, outperforming the baseline by 4.13%.