Audio-Driven Talking Head Generation with Emotion Based on FLAME Geometry Model
摘要
Nowadays, audio-driven talking head generation has gained attention from various fields, yet there are limited studies on generating talking heads with emotions. Existing methods are either limited to generating simple facial expressions only through discrete emotion labels, or rely on complex emotion disentanglement processes of emotion-driven videos. Additionally, due to the inherent limitations of 2D generation models, capturing facial details such as foreheads, wrinkles, and teeth remains challenging. To address these challenges, this paper proposes an emotional talking head generation method based on the FLAME geometric model, which utilizes the geometric structure of the source image to provide 3D priors for the synthesis process. To overcome the emotional expression constraints imposed by emotion labels, the emotional text generated by LLaVA is used to supplement the deficiency of emotions, and a loss function is introduced to calculate the similarity between emotional text and generated images. Meanwhile, the Mamba module is introduced in the decoder to dynamically connect facial details and expressions. Experiments demonstrate that the proposed framework achieves impressive results in image generation details and the expression of complex emotions.