<p>To accurately grasp the transient characteristics of complex gas–liquid flow in axial piston pump, an experimental test system was established for the characteristics of two-phase flow induced vibration in pump. The two-phase flow fluctuation characteristics and its excitation mechanism of pump casing inner wall under different operating conditions were investigated based on a comprehensive computational fluid dynamics model and we are validated by experimental setup. This model elucidates the underlying mechanisms of performance degradation in axial piston pumps by quantitatively linking key operational parameters (pressure and speed) to fluid excitation forces and power loss. It further identifies flow separation and turbulent vortices as the primary sources of suction pressure loss. A variable-density topology optimization method is proposed to redesign the shape of the pump’s suction passage, aiming to minimize pressure loss. This design reduces flow separation and turbulent vortices in the two-phase flow at the suction port, thereby mitigating the fluid excitation force. Results demonstrate that under identical inlet gas–liquid conditions, the optimized suction port achieves a significant reduction in fluid pressure loss. The peak vibration acceleration of the pump is decliend by 20–100 m/s². These findings validate the effectiveness of the proposed method in enhancing hydraulic stability and alleviating gas-induced vibration.</p> Graphical abstract <p></p>