Reconstructing Mel-spectrograms from visual input is a crucial step toward generating intelligible and natural-sounding speech in scenarios where the audio signal is unavailable or compromised. Despite recent advances in Visual Speech Recognition (VSR) for English and other high-resource languages, there is a notable scarcity of datasets and models specifically designed for Spanish. In this work, we introduce a novel Spanish-language audiovisual dataset composed of short video clips of isolated speakers, with aligned transcriptions and their corresponding Mel-spectrogram representations. This work proposes a lightweight convolutional autoencoder model designed to reconstruct segments of Mel-spectrograms directly from sequences of grayscale lip region frames. The architecture processes individual frames through a shared convolutional encoder and combines their latent representations to produce a fixed-size spectrogram patch. Experimental results demonstrate the viability of lightweight, language-specific models for visual-to-audio reconstruction, paving the way for future research in silent speech interfaces and robust speech synthesis for Spanish speakers.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Mel-Spectrogram Reconstruction from Video Lip Sequences Using an Autoencoder Architecture

  • Daphne Sofía González-Cano,
  • Daniel Sánchez-Ruiz,
  • Jesús García-Ramírez

摘要

Reconstructing Mel-spectrograms from visual input is a crucial step toward generating intelligible and natural-sounding speech in scenarios where the audio signal is unavailable or compromised. Despite recent advances in Visual Speech Recognition (VSR) for English and other high-resource languages, there is a notable scarcity of datasets and models specifically designed for Spanish. In this work, we introduce a novel Spanish-language audiovisual dataset composed of short video clips of isolated speakers, with aligned transcriptions and their corresponding Mel-spectrogram representations. This work proposes a lightweight convolutional autoencoder model designed to reconstruct segments of Mel-spectrograms directly from sequences of grayscale lip region frames. The architecture processes individual frames through a shared convolutional encoder and combines their latent representations to produce a fixed-size spectrogram patch. Experimental results demonstrate the viability of lightweight, language-specific models for visual-to-audio reconstruction, paving the way for future research in silent speech interfaces and robust speech synthesis for Spanish speakers.