Planetary geared rotors are widely used in transmission systems due to their compactness and high transmission ratios. Ensuring their reliability requires early fault detection, particularly for gear tooth root cracks, which alter the system vibration signals. These changes in the vibration signals enable the detection of cracks and differentiation of their sizes, allowing for effective fault monitoring. However, real-world planetary geared rotors usually exhibit gear mesh transmission errors, which introduce additional excitations in the system similar to those caused by white noise. This study investigated a planetary geared rotor system under various gear mesh transmission error configurations to identify vibration analysis techniques for detecting gear tooth root cracks. The system dynamic response was analyzed by comparing the acceleration frequency response, power spectrum, and intrinsic mode functions for different sun gear tooth root crack sizes under various gear mesh transmission error conditions. Gear mesh transmission errors significantly hindered the detection of small cracks using these techniques.