<p>Self-excited vibration in a pipe-ball valve system is a complicated dynamic phenomenon that poses severe risks for hydropower plants, including water leakage, penstock rupture, and structural damage. Although this vibration has been correctly attributed to service seal failure in the ball valve, its triggering mechanism remains unknown. This study proposes a novel bidirectional coupling numerical approach to capture the fluid-structure interactions at service seal interfaces. The computational framework employs a dynamic mesh technique to resolve the bidirectional coupling between the transient fluid pressure and seal motion, supplemented by a simplified method of characteristics for efficient simulation of upstream penstock flow. The results show that the pressure ratio of the inlet chamber as well as the friction condition, representing the failure extent of service seal, are the vital factors for triggering the self-excited vibration. Meanwhile, the phase difference between the water hammer wave and service seal displacement creates a self-sustaining oscillation that enables continuous energy transfer from the fluid to the service seal. This energy accumulation process amplifies the vibration amplitude until the system reaches dynamic equilibrium. The vibration of the service seal also induces periodic downstream flow disturbances through the high-velocity jets and vortex formation, with energy transfer dominated by the characteristic vibration frequency. Importantly, the study shows that increasing either the pressure ratio or the friction force effectively suppresses the self-excited vibration, thus providing theoretical foundation for predicting and preventing the self-excited vibration in pipe-ball valve systems, which is particularly important in pumped storage power stations with long penstocks and ball valves located upstream of the pump-turbine units.</p>