<p>This study investigates the Kaup–Broer system, a nonlinear partial differential equation model for weakly nonlinear, long shallow water waves influenced by gravity and surface tension. Through Lie symmetry analysis, we derive five infinitesimal generators of the Kaup–Broer system for capillary waves and construct an optimal system of one-dimensional subalgebras using the commutator and adjoint tables. These symmetries enable similarity reductions that transform the governing system into the system of ordinary differential equations, yielding invariant solutions. Several solutions are visualized in 3D, revealing novel wave structures such as doubly soliton profiles and parabolic cylindrical-like behavior, offering fresh insights into the capillary-gravity wave dynamics. We further discuss the self-adjointness and derive the conservation laws for each generator, reinforcing its integrability and the physical relevance. These new results not only enhance the analytical understanding of the Kaup–Broer system but also highlight its applicability in coastal engineering, fluid transport, nonlinear optics and predicting phenomena, like tsunamis and rogue waves where dispersive and capillary effects are significant.</p>