Aiming at the shortcomings of existing anti-UAV systems in computational resources and small target detection accuracy when using visual sensors to detect UAV targets, an improved small UAV target detection algorithm is proposed. Based on YOLOv11, this algorithm introduce an Iterative Attention-fusion Module (IAM) is introduced to realize context-aware multi-scale feature fusion, enhancing the model’s perception of features at different scales. Depth-wise Convolution (DWConv) is employed in the module to achieve lightweight design without changing feature dimensions. The backbone network design based on Dynamic Convolutions significantly optimizes model parameters while reducing model complexity and avoiding the “low FLOPs” trap. Experimental results show that the proposed model improves mAP@0.5 by 5.2% compared to YOLOv11 and by 6.5% compared to YOLOv12 on the Det-Fly dataset, with a 73.2% increase in model parameters but only a 59.4% increase in FLOPs. Additionally, the model is validated on a self-collected dataset, achieving an mAP@0.5 of 92.4%, demonstrating excellent performance. In conclusion, the model improves small target detection accuracy and reduces model complexity through the improved iterative attention-fusion module, and dynamic convolutions, holding broad application prospects and theoretical value in anti-UAV system applications.