<p>The gear transmission system is a pivotal power transmission subsystem in locomotives, which operates under high load torque and complex external disturbance conditions. Due to neglecting the extended teeth contact caused by gear deformation under load torque, the existing dynamic models of locomotive gear transmission systems lack sufficient precision and have limited dynamic analysis capability. Therefore, this paper proposes an improved dynamic model of the locomotive gear transmission system, which includes the fine gear meshing and locomotive-track coupling models. Notably, the gear meshing model is finely modeled to consider extended tooth contact, gear elasticity, and the structural coupling effect. The motion equations of the wheelset and motor-gearbox affected by extended teeth contact are also established. Then, the established dynamic model of the gear transmission system is experimentally verified. Finally, the effect mechanism of extended teeth contact on the vibration response of the wheelset and motor-gearbox is disclosed. The multi-state mesh, load distribution between teeth, and nonlinear dynamics of the gear transmission system are studied under different locomotive velocities, polygonal wheels, and tooth pitch deviations. The results show that the extended teeth contact will reduce the vibration acceleration of the wheelset and motor-gearbox through the smooth tooth contact force during the alternation process between single-tooth and double-tooth mesh. Both internal and external excitations cause multi-state mesh and nonlinear dynamic behavior in gear transmission systems, including varying degrees of extended teeth contact, tooth disengagement, back-side contact, and phase trajectory expansion. This study offers a theoretical and methodological reference for the comprehensive dynamic evaluation of locomotive gear transmission systems.</p>