In agricultural technology, rapid and accurate diagnosis of plant diseases is crucial for ensuring plant health and improving yield. Currently, automated diagnostic technologies face two main challenges: first, improved accuracy typically comes at the cost of increased model complexity, reducing efficiency and deployability in low-resource settings; Second, models often fail to generalize to complex real-world backgrounds, despite performing well on datasets with simple environments. This dual challenge highlights the need for innovative solutions that balance accuracy with computational efficiency while maintaining robustness across diverse environmental conditions. To bridge this performance-efficiency gap, we developed a lightweight neural network model named Hybrid TopK-Pooling Attention Network (HTPAN). HTPAN uses Inverted Residual Meta Modules (IRMM), blending lightweight convolutions in shallow layers with Hybrid TopK-Pooling Attention (HTPA) in deeper layers. The HTPA mechanism combines pooling with TopK routing local attention to effectively extract critical features, and along with optimization strategies, significantly reduces parameters and computation while preserving high accuracy. We achieved state-of-the-art performance with 99.936% accuracy on the PlantVillage dataset and 77.542% on the PlantDoc dataset, reducing parameter count by 60% and requiring only 0.846G FLOPs. Experimental results demonstrate that our model achieves a strong balance between accuracy and efficiency, making it effective for disease detection in complex and resource-limited environments.