To clear the effect mechanism of self-excited vibration instability on the supercritical tail drive shaft system (STDSS), this paper established the finite element dynamic model of the STDSS, studied the self-excited vibration mechanism, and revealed the effect mechanism of key parameters on self-excited vibration. Results show that the negative damping generated by the larger spline tooth surface friction is greater than the system damping at supercritical speed, leading to self-excited vibration of the system in supercritical conditions. It causes the system to maintain a larger vibration amplitude even after passed the critical speed. At the same time, the system’s response includes the rotational frequency component and the self-excited vibration component near the first critical speed. The degree of self-excited vibration has a direct ratio to the spline teeth number, the spline tooth friction coefficient, the shaft inner diameter, and an inverse ratio to the shaft outer diameter. When there is no O-ring sealing ring, the self-excited vibration amplitude significantly increases, and the unstable speed significantly decreases, which has a significant damping effect. The critical speed and unstable speed of the system are different due to the effect of the shaft material.