Apple leaf diseases can be promptly and accurately identified to improve apple quality and output, modernize apple production, and increase farmers’ financial gains. Regarding the issues of low recognition accuracy, large parameters, and high floating point operations in identifying apple leaf diseases with complex natural backgrounds, we propose an efficient lightweight multi-scale network (ELMSNet) model for apple leaf disease identification. ELMSNet integrates a newly designed multi-scale inverted residual block to fuse disease feature information across different scale spaces, improving the model’s ability to discern diseases while drastically reducing the size of the network model. Additionally, we enhance the efficient multi-scale attention (EMA) module by replacing standard convolution in the large-scale branch with depthwise separable convolution, further achieving lightweight. Then, the improved EMA will be densely embedded into the network model, allowing it to focus more on the key information of the disease. The experimental results indicate that our model has achieved optimal recognition performance on both the PlantVillage dataset with relatively simple backgrounds and the apple leaf disease dataset featuring real farmland backgrounds. It also boasts the lowest parameters and highest throughput among all compared models. The proposed method provides reliable technical support for rapid and accurate identification of apple leaf diseases, holding promise for practical applications in agricultural production. Code and dataset are available at https://github.com/noName0neP/ELMSNet