One-Shot Talking Head Generation with Audio-Aware Identity Compensation
摘要
The primary goal of talking head generation is to synthesize realistic and expressive videos of a person speaking, given an input audio signal and a source image of the person. This involves creating a dynamic, lip-synced, and visually convincing representation of the person in the image as they articulate the provided audio content. But artifacts are shown in generated videos such as blurring of the mouth area, distorted facial features and unstable head and lip motions. The above deficiencies can be attributed to unsync lips and insufficient facial representation and will tremendously diminish the quality of the generated talking head video. To address this issue, we propose a one-shot audio-aware talking head generation architecture, called AaICNet, which is compensated by the learned global facial feature. We use AaICNet to attain lip-sync audio embedding from a random given audio and then drive the portrait to speak, along with the input audio. Specifically, we first develop a audio decoder and face decoder to extract audio feature and face feature and concatenate them into a mixed-feature code. In order to morph the lip movements accurately, we learned a powerful lip-sync discriminator to produce driving video. After the intermediate speaker training stage, we select the person with the highest LSE-C score as the driving image for the next stage of training. Then we introduce an effective compensation module which calculates the global facial structure and prior to enrich the warped source image for the later generation. Extensive experiments demonstrate that our architecture can stably handle the one-shot portrait talking head generation task and can balance the visual quality and the lip-sync accuracy of the generated video.