MaskMel-Prosody-CycleGAN-VC: High-Quality Cross-Lingual Voice Conversion

Voice conversion aims to change the timber of the source speaker to that of the target speaker without changing the speech content. The cross-lingual voice conversion requires non-parallel training data in two different languages, and the prosody and pronunciation of different languages bring challenges to the cross-lingual voice conversion. Previous voice conversion studies based on CycleGAN only used a single pipeline for spectrum mapping. We train two effective pipelines based on the CycleGAN-VC2 separately, in order to get better speech spectrum mapping and speech prosody mapping and decompose F0 (an important prosodic factor) into different time scales by continuous wavelet transform (CWT), so as to better adapt to the hierarchical characteristics of F0 and retain speaker prosody characteristics. In addition, we use MaskMel as the processing features of the spectrum mapping pipeline to improve the sound quality after speech conversion. We also use the self-trained MelGAN to combine F0 features and mel features to synthesize speech again. In this way, we achieve high similarity and naturalness of cross-lingual voice conversion. The MOS test results show that MaskMel-Prosody-CycleGAN framework, we proposed, is superior to the CycleGAN-VC2 baseline in our experiments.