Introduction <p>The roller gear transmission system is an innovative and precise transmission system, featuring long service life, low noise and high reliability.</p> Objective <p>This paper investigates the nonlinear amplitude and frequency characteristics of a new type of precision roller gear transmission system. The aim is to explore the influence of key parameters on the dynamic characteristics of the system.</p> Method <p>Firstly, a two-degree-of-freedom torsional vibration dynamics model was established. The nonlinear control equation including linear contact stiffness and meshing damping was derived using the Lagrange method. Additionally, the amplitude-frequency relationship equation was derived using the L-P method to analyze the effects of short amplitude coefficient (K), damping, external excitation amplitude, rolling radius, and roller length parameters on the amplitude-frequency characteristics of the transmission system.</p> &#xa0;Result <p>The numerical results indicate that when the short amplitude coefficient K value exceeds 0.781, a stability transition determined by the maximum Lyapunov exponent will occur in the system. The experimental vibration spectrum analysis shows that the main component reaches 48 hertz, and the maximum deviation between the theoretical frequency and the measured frequency is 0.4 Hz.</p> Conclusion <p>The experimental vibration spectrum further verified the validity of this model. This model provides practical evidence for the parameter selection and resonance suppression of precision roller gear transmission.</p>