Purpose <p>To explore the impact of gear floating on the system's nonlinear dynamic characteristics, a gear floating model was developed based on the concept of gear floating.</p> Methods <p>A nonlinear dynamic model, bending-torsional-axial-pendular (BTAP), has been developed for a coaxial reverse closed differential herringbone gear transmission system (CRCDHGTS), accounting for gear floating. This model considers factors such as gear floating backlash, tooth surface friction, gyroscopic effects, time-varying meshing stiffness (TVMS), meshing damping, and dynamic meshing parameters. A calculation model for the floating backlash and floating TVMS of a herringbone gear system was derived, and the nonlinear dynamic response of the gear system was solved using the Runge–Kutta method.</p> Results <p>The influence of input speed, initial backlash, gear float value, and system transmission error on the nonlinear dynamic vibration characteristics is analyzed using various diagrams, including bifurcation diagrams, maximum Lyapunov exponent (MLE) plots, time history diagrams, frequency diagrams, phase diagrams, and Poincaré section diagrams.</p> Conclusions <p>The research reveals that gear floating diminishes the chaotic motion behavior of the system under different excitation factors, thereby improving the system's global bifurcation characteristics. The developed BTAP coupled nonlinear dynamic model provides more accurate numerical solutions compared to models with fixed meshing parameters, rendering it more suitable for analyzing the system's dynamic characteristics. Analysis of the gear floating value indicates an optimal range of 0–20&#xa0;μm and 34–43&#xa0;μm for generating periodic motion, with floating values around 10–20&#xa0;μm demonstrating better performance in mitigating the negative effects of initial backlash and transmission error.</p>