Gear meshing and rolling bearings exhibit pronounced nonlinear characteristics in gearbox systems, attributed to contact state variations. Research simultaneously considering the dynamic interaction between gear meshing and bearings to explore gearbox vibration characteristics remains limited. Thus, this paper proposes a nonlinear bearing model by extending the traditional gear dynamics model, incorporating dynamic bearing forces, rotational speed, variations in the number of rolling element contacts, and bearing clearance. The vibration responses of the gearbox system under the dynamic interaction between gear meshing and bearings are investigated. The dynamic variation characteristics of bearing stiffness and the impacts of bearing nonlinear factors on the dynamic responses of the gear pair, bearings, and housing are analyzed.Numerical results show that gear meshing and high-speed centrifugal force significantly affect the dynamic stiffness of bearings. Additionally, the nonlinear excitation of bearings increases the vibration of dynamic transmission error, bearings, and the housing. The predicted vibration acceleration of the housing, when bearing nonlinearity is considered, is closer to the experimental value.the model proposed in this paper has remarkable engineering significance and can assist engineers in understanding the characteristics and effects of vibrations induced by bearing nonlinearity.