The widespread application of self-excited induction generator (SEIG) in stand-alone renewable generation systems accelerate construction of new type power systems. Based on the state-space model of SIEGs in two-phase stationary reference frame, the paper proposes a theoretical analysis approach to motion stability using the Nyquist frequency-domain stability criterion. An equivalent open-loop frequency characteristics model is established according to the characteristic equation of the system. Then, the necessary and sufficient conditions of the general crossing frequency are obtained through motion stability analysis by the Nyquist frequency-domain criterion. Satisfying both conditions simultaneously is the stator frequency. Analytical calculations can directly provide the critical values of the rotor speed under given exciting capacitance, resistance, and magnetizing inductance and the necessary condition for each given value, as well as the critical values of the exciting capacitance under given rotor speed, resistance, and magnetizing inductance. Computation cases for the initial self-excitation buildup transient is performed on the prototype and proves the correctness, validity, and generality of the proposed theoretical analysis approach.