In recent years, emotional voice conversion and expressive speech synthesis have gained attention due to their applications in areas such as automated dubbing, human-computer interaction, and assistive technologies. Our research proposes an AI-based dubbing system, SyncVox, which presents a seamless voice dubbing custom pipeline designed to provide seamless voice conversion across languages. It addresses low-resource video dubbing using various advanced technologies like speech recognition, translation, and style transfer. The pipeline combines speaker embeddings with advanced techniques to produce natural-sounding, speaker-alike voice synthesis. By employing multitask learning with Text-To-Speech, the pipeline is capable of capturing rich linguistic information while retaining languages; this allows content creators to dub videos without compromising the original speaker’s intent and naturalness. Early results show that this system effectively synthesizes natural-sounding speech with high emotional fidelity.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

SyncVox: Synchronized AI Based Video Dubbing

  • Owais Ansari,
  • Hemangini Patel,
  • Tejas Maroo,
  • Morvi Panchal,
  • Nikita Raichada

摘要

In recent years, emotional voice conversion and expressive speech synthesis have gained attention due to their applications in areas such as automated dubbing, human-computer interaction, and assistive technologies. Our research proposes an AI-based dubbing system, SyncVox, which presents a seamless voice dubbing custom pipeline designed to provide seamless voice conversion across languages. It addresses low-resource video dubbing using various advanced technologies like speech recognition, translation, and style transfer. The pipeline combines speaker embeddings with advanced techniques to produce natural-sounding, speaker-alike voice synthesis. By employing multitask learning with Text-To-Speech, the pipeline is capable of capturing rich linguistic information while retaining languages; this allows content creators to dub videos without compromising the original speaker’s intent and naturalness. Early results show that this system effectively synthesizes natural-sounding speech with high emotional fidelity.