<p>Planetary gearboxes are widely used due to their excellent load-carrying capacity. However, the unique operation mode and meshing relationship of planetary gears have produced complex vibration signal modulation phenomena, which significantly distinguish the motion and transmission characteristics of planetary gears from those of conventional fixed-axis gear pairs. Therefore, the commonly used fault diagnosis methods for fixed-axis gear pairs are difficult to be directly applied to the fault diagnosis of planetary gears, which increases the difficulty of fault diagnosis in planetary gearboxes. Vibration signal separation (VS) technology is used to extract the vibration signal of planetary gears from the original signal, but this technology lacks the key step of determining the initial phase of separation, thus hindering its application in engineering practice. This study mainly makes improvements in two aspects: initial phase selection, signal matrix reconstruction and compensation, to enhance the effect of vibration signal separation technology. Specifically, it includes: 1) Optimizing the initial phase selection method and using the time-domain synchronous averaging (TSA) technology to process the vibration signal with the planetary carrier’s rotational frequency as the period; 2) Resonance demodulation technique is used to enhance the fault signal components in the original vibration signal; 3) Improve the signal reconstruction method, compensate for the selected initial phase based on the parameters of the planetary gearbox, obtain the vibration signal generated by the meshing of the planetary gear with other components, and screen out the vibration separation signal with the best signal-to-noise ratio. Finally, the improved method was verified through experiments, and the signal quality was evaluated by the signal-to-noise ratio. The verification results show that the quality of the vibration separation signal obtained by the improved method has been enhanced.</p>