SyncVox: Synchronized AI Based Video Dubbing
摘要
In recent years, emotional voice conversion and expressive speech synthesis have gained attention due to their applications in areas such as automated dubbing, human-computer interaction, and assistive technologies. Our research proposes an AI-based dubbing system, SyncVox, which presents a seamless voice dubbing custom pipeline designed to provide seamless voice conversion across languages. It addresses low-resource video dubbing using various advanced technologies like speech recognition, translation, and style transfer. The pipeline combines speaker embeddings with advanced techniques to produce natural-sounding, speaker-alike voice synthesis. By employing multitask learning with Text-To-Speech, the pipeline is capable of capturing rich linguistic information while retaining languages; this allows content creators to dub videos without compromising the original speaker’s intent and naturalness. Early results show that this system effectively synthesizes natural-sounding speech with high emotional fidelity.