As UAVs become increasingly prevalent in urban security monitoring, they are confronted with the challenge of accurately identifying small targets that are easily obscured by complex backgrounds, dense buildings, and dynamic pedestrian flows. In response to these challenges and the demands of real-world applications, we introduce SMT-Net, a system tailor-made for UAV patrolling. SMT-Net marries the Compact Axial Transformer Block with Scale-Adaptive Modulation, striking an effective balance between detection precision and computational expense. The Compact Axial Transformer Block comprises two innovative components: Compact Axial Attention and Fine-grained Feature Enhancement. Compact Axial Attention reduces parameter count and model intricacy while preserving crucial feature information. Concurrently, the introduced Fine-grained Feature Enhancement substantially boosts the model’s capability to apprehend target details, thereby enhancing classification and detection efficiency for diminutive objects. The Scale-Adaptive modulation adeptly seizes semantic information across disparate feature strata, augmenting the detection acuity for minuscule targets. Furthermore, to improve boundary precision in small object detection, we introduce the shape-IoU method, enhancing detection accuracy. On our designed DRP-Dataset for UAV road patrolling imagery, SMT-Net achieved an outstanding 88.0 \(\%\) mAP, particularly demonstrating remarkable superiority in small object detection, and outperforming all mainstream methodologies. The experiments substantiate that SMT-Net can satisfy the stringent demands for accurate and efficient detection across various UAV platforms in diverse complex scenarios.