In today’s rapidly developing emerging technologies, people have put forward higher requirements for efficient mobile device computing power mobile devices face problems such as high computational and storage costs. This article addressed the redundancy issue in the convolutional neural network (CNN) model, using pruning methods to remove redundant parameters and greatly reduce model size. By applying quantization methods, the parameters in the model were compressed from floating-point form to low integers, thereby reducing the computational load and memory overhead of mobile devices. Training large-scale teacher networks to gain more knowledge ensured the accuracy of the model. The experimental results showed that the optimized model maintained high accuracy (from 95.0 to 93.5%, still within an acceptable range). Through the research in this article, an efficient convolutional compression and acceleration method can be provided for the application of deep learning on mobile terminals, which further improves the performance of mobile terminals in intelligent applications such as real-time image processing and speech recognition.