With the continuous development of agricultural production technology, there is a growing demand for pest identification and control. Accurate identification of pests is the key to realize intelligent crop protection and sustainable agricultural development. However, due to the ability of insect metamorphosis and mimicry hiding, real-world pest identification is still challenging. This study introduces a variant of the MobileViT model designed for fine-grained pest recognition in agriculture. Specifically, we embed the SimAM attention mechanism into intermediate MobileNetV2 layers to enhance spatial feature discrimination, and design a Two-Stage Fully Connected Classifier Head (Two-Stage FC Head) with dropout and batch normalization to promote generalization. We evaluate our model on the large-scale and highly imbalanced IP102 dataset, which comprises 102 pest species. The enhanced model reaches a top-1 accuracy of 69.02%, exceeding the baseline MobileViT-S while preserving low inference latency. Moreover, our method surpasses classical CNNs in accuracy. The results demonstrate that integrating lightweight model with attention mechanisms and enhanced classifiers significantly improves performance for agricultural pest recognition while preserving the feasibility of deployment on edge devices. This work demonstrates strong potential for practical deployment in AI-powered agricultural systems, supporting intelligent pest classification in resource-constrained real-world environments.