With the rapid development of human‒computer interaction technology, gesture recognition systems have been widely recognized as critical technologies for improving the naturalness and efficiency of interactions, and they play essential roles in industrial production, scientific research, and so on. In actual applications, existing gesture mouse control systems often lack accuracy and delay in response, failing to meet demand. To solve this problem, we propose a gesture control mouse system based on MobileNetV3-Efficient Channel Attention-Ghost Yolov5 by replacing the YOLOv5 backbone network, Cspdarknet53, with the lightweight MobileLeNetv3 and integrating ECA and GhostNet into the backbone network. The results showed that this technique could identify gestures effectively and accurately and had high processing speed. The improved gesture control mouse system surpassed the existing technology in nine experimental environments. The model execution time was reduced by 8%, and the recognition accuracy was improved by 4%. These advantages provide more opportunities to improve the user experience, improve production efficiency, and solve practical problems.