Object detection is crucial in various Unmanned Aerial Vehicle (UAV) applications, including security monitoring, search and rescue operations, and military surveillance. However, objects observed from the drone view frequently exhibit the characteristics of small size and complex shapes. Current small object detection methods often encounter issues of missed detections and false positives. In this work, we propose a lightweight multi-scale small object detection network for UAV perspective (UAV-LMDN) to enhance the detection accuracy of small objects while reducing the model’s parameters. Firstly, we propose a lightweight multi-scale detection head structure, enabling the detection of objects with smaller pixel sizes and reducing the network parameters by 1) adding a small object detection head that integrates shallow feature layers and 2) removing the large object detection head and its associated network layers. Secondly, we introduce the efficient channel attention module at the neck of the network, enhancing the ability to capture key details for small objects. Experimental results demonstrate that our method achieves a 2.9% improvement in mean Average Precision (mAP) on the VisDrone dataset and a 5.1% improvement on the TinyPerson dataset. Furthermore, our method reduces the parameters by 33.5% compared to the baseline method.