The rapid development of drone technology has made visual detection of unfriendly drones a hot topic. However, convolutional neural networks have not fully leveraged their advantages in tiny drone detection tasks, and current methods have failed to balance detection precision and inference latency. Due to the small size of drones, weak visual features are easily lost during the flow of information, and expensive computing and storage resources of edge devices, drone detection remains a challenge. To address this challenge, we develop a lightweight partial multi-layer aggregation network named LPMANet. In this network, a slim cross channel interaction module SCCI block is proposed. By introducing partial convolution and attention mechanisms, channel information interaction is guided at a lower cost while enhancing the visual features of drones, and SCCI block is embedded in multi-layer aggregation LPMA block to alleviate gradient information redundancy. We also exploit a novel progressive feature fusion method to enhance the semantic information of shallow features. In addition, we use Unreal Engine 5 to simulate over 8000 drone images in different scenes and weather conditions and provide corresponding annotated object bounding boxes as a drone benchmark dataset. Experimental results show that LPMANet achieves state-of-the-art results compared to existing methods on simulated datasets and public dataset TIB-Net, and it demonstrates outstanding real-time performance on the HUAWEI Ascend 310 chip hardware platform.