The object detection method based on machine learning has achieved remarkable results. High storage and computing resources cannot be provided. Traditional video object detection algorithms have complex network structures and high computing hardware requirements. Therefore, it is necessary and practical to design a video object detection algorithm with high real-time performance and accuracy. The video object detection algorithm is improved as follows from the aspect of increasing detection speed: In this paper, fine-grained pruning based on weight sharing matrix was proposed to optimize the weight sharing matrix, select appropriate regularization methods, and determine the optimal pruning rate and the number of fine tuning through comparative experiments to reduce the number of parameters by 33% and significantly reduce the number of network parameters. In this paper, through the neural network system, under the video equestrian club monitoring system, this algorithm can recognize people's faces, to identify people in and out. With 98.7% accuracy achieved through deep neural networks, the system should be widely used in clubs in the future.