AuraVox: Redefining Communication with End-to-End High-Fidelity Visual-to-Speech Synthesis Powered by Transfer Learning
摘要
The Aura Vox system introduces a cutting-edge visual-to-speech synthesis framework that converts lip movements into intelligible, natural-sounding speech with exceptional accuracy. The model employs a comprehensive deep learning framework, utilizing convolutional neural networks (CNNs) to extract visual features and recurrent neural networks (RNNs) to analyze temporal patterns. High-quality audio synthesis is achieved through Tacotron 2 for spectrogram generation and HiFi- GAN for real-time high-fidelity audio output. A widely used subjective evaluation metric, MOS assesses perceived quality based on user feedback with a score of 4.3, a word error rate(WER) of 13.3%, and a character error rate (CER) of 4.4%. Enhanced by transfer learning, Aura Vox adapts effectively to diverse speech patterns and challenging environmental conditions, making it a powerful tool for accessibility applications, such as helping people with speech disabilities or in audio-deprived environments.