<p>Composite flywheels are used in large-capacity flywheel energy storage due to their high strength and high energy storage density. We studied the instability of the composite flywheel rotor system caused by internal damping. First, considering the gyroscopic effect, ply characteristics, and internal damping of the composite material, the dynamic model of the composite rotor was derived based on the simplified homogenized beam theory (SHBT). The stability of the rotor system was determined by the Lyapunov stability criterion. The effects of ply, rotor shape, and support parameters on the stability of the system were studied. The following conclusions were drawn: The ply thickness and sequence, positive and negative ply angle, the length of upper bearing shaft, middle shaft, lower bearing shaft and shell support parameter have little effect on the threshold speed. The other parameters have great influence. The threshold speed is negatively correlated with the average ply angle, the thickness of the composite rim, and the inner-to-outer diameter ratio of the rotor. The threshold speed is greatly affected by the rotor stiffness and damping within a certain range and is more sensitive to the lower bearing parameters.</p>