Purpose <p>To enhance the reliability and lifespan of downhole turbine generators operating in the oil well drilling process, this study addresses the mechanical vibration by proposing a novel viscous damper, providing a customized design solution and a basis for parameter optimization to control vibration.</p> Methods <p>The damper operates based on viscous damping, where rotor vibrations drive damping oil through precision orifices to dissipate energy, enabling adaptation to harsh downhole conditions without requiring an external oil supply. The influence of structural parameters on damping performance was investigated using a nonlinear damping model, computational fluid dynamics simulations (CFD), and orthogonal testing. A comprehensive shaft system dynamics model combining simulation and experiment was developed to evaluate the damper's effects on excitation responses, bearing dynamic loads, and rotor critical amplitude.</p> Results <p>Results demonstrate that structural parameters significantly affect damping force trends, with damping orifice diameter identified as the dominant factor. Shaft system simulations indicated that the damper reduces bearing dynamic loads and the maximum rotor critical amplitude by 24.5%. Experimental validation confirmed a 25% reduction in harmonic excitation amplitude; however, increased rotor amplitude was observed under transient impact loads due to stiffness-damping interaction between the damper and rotor.</p> Conclusions <p>The proposed viscous damper effectively mitigates vibrations in downhole rotating machinery and offers a valuable foundation for the design and parametric optimization of turbine generator vibration control systems, though its performance under sudden impact loads necessitates further investigation.</p>