<p>When the planetary transmission system operates under poor lubrication conditions, it is prone to gear pitting, which will seriously affect the operational stability of the system. Therefore, it is of great theoretical value and engineering significance to study the vibration characteristics of the system under different pitting degrees. In this study, a nonlinear dynamic model considering friction, time-varying meshing stiffness and clearance under pitting corrosion and transmission error is established for planetary transmission system. Based on the lumped mass method, the nonlinear vibration differential equation of the system is established, and the Runge–Kutta algorithm is used for numerical solution. The nonlinear dynamic behavior of the system is systematically studied by means of bifurcation diagram, time domain diagram, phase plane diagram, wavelet time–frequency analysis and Lyapunov exponent. In addition, the influence of meshing damping, meshing stiffness and external load fluctuation on the primary resonance characteristics of the system is analyzed by multi-scale method. In order to verify the reliability of the model, the fault simulation experiment of planetary gear set is carried out. The results show that under the action of external excitation frequency, the system presents a complex vibration state with periodic motion and chaotic motion alternately, which reveals the global dynamic evolution law of the transmission system under the action of external excitation frequency. The pitting corrosion of gear will significantly reduce the stability of the system, and with the increase of excitation frequency, the influence of pitting corrosion on the dynamic characteristics of the system is more significant.</p>