ViT-GA-AVSS: Optimizing Vision Transformer-Based Audio-Visual Speech Synthesis Approach Using Genetic Algorithm
摘要
Audio-Visual Speech Synthesis (AVSS) plays a crucial role in generating synchronized audio-visual outputs of a target speaker from the speech input of a source speaker, while preserving linguistic content. The existing research on AVSS in the literature primarily focus on developing advanced models for the enhanced audio-visual outputs. However, optimizing AVSS models remains challenging due to the complexity of hyperparameter selection and network architecture design. To address these limitations, we propose a Genetic Algorithm (GA)-driven optimization framework that jointly performs hyperparameter tuning and Neural Architecture Search (NAS) to enhance a Vision Transformer (ViT)-based autoencoder (AE) for AVSS. GA efficiently explores the hyperparameter space, optimizing key factors such as patch size, attention heads, embedding dimensions, and learning rate, while simultaneously guiding NAS to refine the ViT-based architecture by dynamically adjusting network depth, attention mechanisms, and layer configurations. This GA-optimized AVSS model is evaluated on benchmark datasets, including VoxCeleb2, LRS3-TED, and VCTK, demonstrating superior performance compared to state-of-the-art (SOTA) methods. The experimental results show significant improvements in speech intelligibility, synchronization, and synthesis quality, validating the effectiveness of evolutionary search strategies in AVSS. The proposed approach offers a scalable and adaptive solution, reducing manual effort while enhancing robustness across diverse speech conditions, and establishing the GA-driven hyperparameter and architecture optimization as a powerful technique for advancing AVSS, with applications in virtual avatars, assistive communication, and multimedia content creation.