Abstract <p>The nonlinear dynamic behavior of the planetary gear system will be altered as the gear failure occurs. Tooth root crack are among the most prevalent forms of gear failures, leading to a degradation in meshing stiffness and consequently having a substantial impact on the nonlinear dynamic characteristics of the system. In this paper, a comprehensive dynamic model of a planetary gear system with tooth root crack is established to investigate the nonlinear dynamic behavior of the system in a faulty state. Firstly, considering the elastoplastic deformation of the cracked gear tooth and combining it with the potential energy method, we derive an accurate meshing stiffness for the gear pair. Secondly, by introducing time-varying tooth backlash, time-varying meshing stiffness, damping, and manufacturing errors, we establish a nonlinear dynamic model for the planetary gear system. Finally, we compare the global bifurcation characteristics between a healthy system and a faulty one under different excitation frequencies and tooth backlash to reveal the path of the system emerging into chaos. The results demonstrate that faulted planetary gear systems are more prone to enter chaotic states with nonlinear excitation parameters. This research can offer a robust theoretical framework for the design and fault diagnosis of gearboxes.</p>