HyperLips: hyper control lips with high resolution decoder for talking face generation

Talking face generation has a wide range of potential applications in the field of virtual digital humans. However, rendering high-fidelity facial video while ensuring lip synchronization remains challenging for existing audio-driven talking face generation approaches. To tackle this problem, we present HyperLips, a two-stage framework comprising a hypernetwork for lip control and a high-resolution decoder for generating high-fidelity faces. In the first stage, we develop a base face generation network that utilizes the hypernetwork to regulate the encoding latent code of the visual face information based on audio. The process begins using FaceEncoder to extract features from the face frame, resulting in a latent code. Next, HyperConv modifies the latent code to synchronize the lip movement with the audio, with its weighting parameters updated by HyperNet using the audio features as input. Finally, the FaceDecoder decodes the modified and synchronized latent code into visual face content. In the second stage, higher-quality face videos are obtained through a high-resolution decoder. To enhance face generation quality, we trained a high-resolution decoder, HRDecoder, using face images and detected sketches generated from the first stage as input. The experimental results on benchmarks LRS2 show that the performance of HyperLips is obviously beyond the state-of-the-art methods, e.g., the PSNR, SSIM (higher is better) are improved from 33.281, 0.894 to 34.914, 0.914, respectively; LPIPS, FID, LMD (lower is better) are reduced from 0.051, 5.199, 1.471 to 0.032, 3.690, 1.186, respectively. Our method outperforms state-of-the-art with more realistic, high-fidelity, and lip-synchronized talking faces, as demonstrated by extensive quantitative and qualitative experiments. Please refer our project page as below:https://semchan.github.io/HyperLips_Project/