<p>Excessive vibration in marine propulsion gearboxes critically impacts reliability, noise pollution, and ship stealth. This study presents a novel rigid-flexible coupling dynamic model for a double-layer casing herringbone planetary gearbox with journal bearings, integrating experimentally determined dynamic characteristics of the vibration isolator into a reduced-order flexible casing model. A thorough examination of the casing’s modal vibration coupling mechanism enabled the development of a comprehensive rigid-flexible coupling dynamic model for the gearbox. This model facilitated evaluation of the vibration isolation performance across varying load torques and input speeds. The results reveal the favourable modal decoupling ratio in the translation direction, with the first six orders predominantly contributing to the modal behaviour. Furthermore, the vibration isolation performance of the gearbox varies with changes in the meshing frequency, and it exhibits the negative correlation with the load torque while showing the non-monotonic relationship with the input speed. Experimental and simulation validation via vibration level differences confirm effective isolation. This research enhances marine gearbox reliability and noise reduction.</p>