ZSEmo-MTVITS: A Zero-Shot Cross-Lingual Emotional Speech Synthesis Model for Mandarin and Tibetan Based on VITS
摘要
Cross-lingual speech synthesis is an active research area in speech synthesis. However, the intricate interplay of linguistic, emotional, and timbral features in cross-lingual scenarios poses significant challenges, often resulting in non-native accents and limited emotional expressiveness in generated speech. These issues are particularly pronounced in low-resource languages due to the scarcity of high-quality datasets and specialized modeling techniques. To address these challenges, we propose ZSEmo-MTVITS, a zero-shot cross-lingual emotional speech synthesis model for Mandarin and Tibetan based on the VITS. We constructed a novel Tibetan emotional speech dataset containing 2500 samples of the Lhasa dialect, covering five emotions (angry, happy, neutral, sad, and surprised), and evaluated Tibetan emotional dataset using the established end-to-end model VITS. Furthermore, we optimize the original VITS model by introducing extra emotion, language, speaker labels and a pre-trained timbre converter model to enable zero-shot cross-lingual emotional speech synthesis for Mandarin and Tibetan. The experimental results show that the ZSEmo-MTVITS model outperforms the baseline model in subjective evaluations, demonstrating superior emotional speech synthesis performance. The model effectively achieves Zero-Shot cross-lingual emotional speech synthesis for Mandarin and Tibetan.