In view of the lack of lightweight algorithms specially designed to be deployed on edge devices in laboratory environment to detect mouse behavior, this paper proposes a lightweight mouse key behavior detection model named YOLOv8n-PGE, based on the YOLOv8n model. Firstly, A lightweight and efficient feature extraction module is designed, which utilizes dual attention and partial convolution. This module significantly reduces the number of model parameters and computations while enhancing the model’s ability to extract critical features. Secondly, an auxiliary learning branch is introduced, which enhances the model’s learning capability without increasing the number of parameters or computations, thus addressing the bottleneck issues caused by network deepening. Experimental results show that the proposed YOLOv8n-PGE model not only reduces the number of parameters and computations but also improves accuracy, meeting the requirements for real-time behavior detection of mice in laboratory settings.