The gear surface was prone to form periodic regular ripples in worm wheel grinding, which led to an increase in the main-order vibration during the End-of-Line (EOL) detection. Vibration was transmitted to the gear surface in the form of ripples; however, the importance of wheel vibration had been overlooked in previous studies on the optimization of gear grinding process parameters. Therefore, this paper proposed an optimization method of process parameters considering the influence of grinding wheel vibration displacement. Firstly, the experiment was carried out with the self-developed grinding data acquisition system, and the Spearman coefficients and the Kernel Principal Component Analysis (KPCA) method were utilized for the feature selection; Then, a predictive model for main order vibration was established based on BP network and the BP network was optimized through Genetic Algorithm (GA); Finally, an optimization model for process parameters based on Improved Particle Swarm Optimization (IPSO) was established, with the prediction model serving as the fitness function. In the practical application, the optimized combination of process parameters resulted in a 49.52% reduction in the main order vibration.