AmoMimic: Speech-Driven 3D Face Based on Emotion Enhancement and Multimodal Approach
摘要
Speech-driven 3D facial animation has made some progress in achieving natural lip movements, but existing methods still fail to effectively capture the delicate emotions conveyed in speech, and the generated facial movements often appear single and lack variation. These limitations result in facial animations appearing stiff and repetitive, thereby reducing user immersion and limiting their widespread application. To address the issues of insufficient emotional expression and difficulty in multi-scale motion modeling, this paper proposes the AmoMimic framework for generating speech-driven 3D facial animations with emotional expressiveness. This method achieves collaborative optimization of emotional expression and lip sync by jointly modeling the emotional features of text and speech, combined with layered motion coding technology. This article designs a joint emotion embedding Amo module to generate high-quality emotion features, and an EmoVAVAE module as a motion prior for generating driving facial motion parameters. The experiment shows that this method outperforms existing methods in various indicators, verifying the effectiveness of joint emotion embedding and hierarchical motion modeling, and achieving a balanced optimization of emotional expressiveness and mouth shape accuracy. Through multimodal emotion fusion and layered motion decoupling, this method breaks through the bottleneck of balancing emotional richness and motion authenticity in traditional speech driven animation.