The Aura Vox system introduces a cutting-edge visual-to-speech synthesis framework that converts lip movements into intelligible, natural-sounding speech with exceptional accuracy. The model employs a comprehensive deep learning framework, utilizing convolutional neural networks (CNNs) to extract visual features and recurrent neural networks (RNNs) to analyze temporal patterns. High-quality audio synthesis is achieved through Tacotron 2 for spectrogram generation and HiFi- GAN for real-time high-fidelity audio output. A widely used subjective evaluation metric, MOS assesses perceived quality based on user feedback with a score of 4.3, a word error rate(WER) of 13.3%, and a character error rate (CER) of 4.4%. Enhanced by transfer learning, Aura Vox adapts effectively to diverse speech patterns and challenging environmental conditions, making it a powerful tool for accessibility applications, such as helping people with speech disabilities or in audio-deprived environments.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

AuraVox: Redefining Communication with End-to-End High-Fidelity Visual-to-Speech Synthesis Powered by Transfer Learning

  • Krishna Balu Somalaram,
  • Shashidhar V. Reddy,
  • P. S. Sanjay,
  • V. Nandan,
  • B. Ashwini

摘要

The Aura Vox system introduces a cutting-edge visual-to-speech synthesis framework that converts lip movements into intelligible, natural-sounding speech with exceptional accuracy. The model employs a comprehensive deep learning framework, utilizing convolutional neural networks (CNNs) to extract visual features and recurrent neural networks (RNNs) to analyze temporal patterns. High-quality audio synthesis is achieved through Tacotron 2 for spectrogram generation and HiFi- GAN for real-time high-fidelity audio output. A widely used subjective evaluation metric, MOS assesses perceived quality based on user feedback with a score of 4.3, a word error rate(WER) of 13.3%, and a character error rate (CER) of 4.4%. Enhanced by transfer learning, Aura Vox adapts effectively to diverse speech patterns and challenging environmental conditions, making it a powerful tool for accessibility applications, such as helping people with speech disabilities or in audio-deprived environments.