Purpose <p>This study aims to investigate the nonlinear dynamic vibration characteristics of spiral bevel-planetary gear coupling transmission systems, particularly focusing on their behavior under varying support stiffness. It seeks to address vibration control and noise reduction challenges in such systems used in high-precision and extreme environments, providing a theoretical basis for performance optimization and stability improvement.</p> Methods <p>A 3D model and nonlinear dynamic mathematical model of the transmission system are established, incorporating backlash, sliding friction, time-varying mesh stiffness, mesh damping, and comprehensive transmission errors. A dynamic characteristic test platform is built for validation. The system's global dynamic behavior is analyzed using time-frequency domain methods, phase diagrams, Lyapunov exponents, wavelet transforms, and bifurcation diagrams.</p> Results and Conclusions <p>The nonlinear model reveals that support stiffness induces alternating periodic-chaotic motion states in the system. The optimal range of dimensionless support stiffness is determined, which avoids resonance and chaos, enhancing operational stability. Experimental validation confirms the model's effectiveness. This research lays a foundation for vibration prediction, performance analysis, and optimal design of such transmission systems.</p>