Purpose <p>The purpose of this paper was to analyze the evolution law and dynamic characteristics of the helical gears’ coexistence response considering the transition of meshing state, and to check the strength of the teeth when the coexistence response occurred.</p> Methods <p>The time-varying geometric parameters, friction coefficient, time-varying meshing stiffness and load distribution coefficient of positive meshing and reverse impact were analyzed respectively. The dynamic model of helical gear considering de-meshing and reverse impact was established. The global bifurcation diagram, cell mapping method, phase diagram and Poincaré section were used to analyze the evolution of coexistence response and dynamic characteristics with the parameters of helical gear system.</p> Results <p>The failure conditions were established, according to the dynamic meshing force solved by the helical gear dynamic model, the tooth surface contact fatigue strength, tooth root bending fatigue strength and plastic deformation fatigue strength of the meshing tooth pair were checked when different responses occurred.</p>