Purpose <p>Purpose Focusing on the pitting faults problem of helical planetary gear systems and conduct dynamic modeling and analysis.</p> Methods <p>This work proposes a time-varying meshing stiffness (TVMS) calculation method applicable to the external and internal gear pairs, simplifies the actual pitting shape into an elliptical pit, and establishes the analytical model for the external and internal gears in healthy and pitting states. The effects of pitting fault at varying levels of damage on the TVMS of each gear pair are analyzed, and the influence law of pitting fault on the periodic characteristics of TVMS is revealed by considering the meshing phase. A 24-degree-of-freedom (DOF) dynamics model for helical planetary gear systems is established, and the TVMS results is introduced into the model. The vibration response of the sun gear, planetary gear, and ring gear under different pitting damage levels is obtained by numerical simulation, and the impact characteristics in the time-domain signal are analyzed and modulation phenomena in the frequency-domain signal. Finally, the accuracy of the established dynamics model is verified by the vibration acceleration signals measured on a test rig.</p> Results <p>The constructed pitting faults dynamics model can accurately simulate the TVMS changes of the planetary gear system, and reveal the mapping relationship between the degree of fault and the vibration characteristics.</p> Conclusion <p>The dynamic model established in this paper can comprehensively and effectively analyze the vibration response characteristics of planetary gear systems with pitting faults. It provides an effective reference for the dynamic characteristic analysis, fault feature extraction, and diagnosis of helical gear planetary systems.</p>