Purpose <p>In engineering applications, floating breakwater mooring systems often face complex dynamic responses caused by wave height changes. Therefore, a method is needed to accurately analyze the influence of wave height changes on the dynamic response of mooring systems in order to understand their dynamic behavior under different sea conditions.</p> Methods <p>Based on the Absolute Nodal Coordinate Formulation (ANCF), the mass matrix, stiffness matrix, and generalized elastic force matrix for the mooring line elements are derived in detail, while also accounting for Morison’s current force, seabed contact force, and the external loads exerted by the floating box on the mooring system. Using the Lagrangian equation, the dynamic equations are then established.</p> Results <p>The effectiveness and accuracy of the method are verified by three cases: mooring cable fixed float model, flexible single pendulum model and sequential co-simulation. In addition, the influence of wave height variation on dynamic response of floating breakwater mooring system is revealed, including the tension variation in the cable, the relationship between system motion and wave.</p> Conclusion <p>The mooring line on the wave facing side bears more tension when the wave height changes, and the displacement of the middle node changes more significantly. In addition, the proposed sequential co-simulation method can effectively simulate these nonlinear behaviors and accurately reproduce the dynamic characteristics of the system.</p>