As mechanical equipment develops towards high reliability, high speed and silence, the vibration and noise of gear transmission systems have attracted more and more attention. Aiming at the vibration and noise problems generated by gearboxes during operation, the dynamic model of the gear-shaft-bearing-gearbox coupling system is established based on the dynamics and structural acoustics of the gear transmission system. The dynamic force of the bearing is calculated through advanced whine analysis as the excitation force of the gear structure system. The harmonic response analysis of the gearbox is carried out using the modal superposition method in finite element analysis to solve the vibration response of the gearbox. The acoustic boundary element mesh model of the gearbox is established using the boundary element method. The results of the harmonic response analysis are used as boundary conditions to predict the vibration and noise characteristics of the gearbox. Finally, the gear transmission system test device is established through the FZG gear testing machine, the vibration signal of the gearbox is collected, and its main frequency components are analyzed and compared with the simulation results. The results show that this method can effectively predict the vibration and noise problems of the gear transmission system, and the accuracy of the theoretical simulation is verified, which provides a reliable basis for the noise control and structural optimization of the gear transmission system.