FusionCraft: Fusing Emotion and Identity in Cross-Modal 3D Facial Animation

Recent advancements in speech-driven 3D facial animation have shown promising progress, yet authentically conveying intricate expressiveness, especially in emotions and individual identity, remains challenging. Many studies focus on lip synchronization, overlooking emotional subtleties and personal uniqueness. To address this gap, we introduce a novel method to generate 3D facial expressions that resonate deeply with both emotion and identity, guided by speech and user prompts. Our innovation lies in an emotion-identity fusion mechanism—a pre-trained self-reconstruction codebook derived from diverse emotional facial movements, serving as a benchmark for expressive motion. Prompt words evolve into facial representations capturing emotion and identity, projected onto 3D templates. Harmonized with speech audio and a specified emotion, our algorithm animates a 3D avatar, reflecting intended emotion and unique identity. Our model’s effectiveness is enhanced by advanced autoregression, uniting emotion and identity through feature fusion module and a tailored loss function. Thus, our approach is a robust tool for crafting 3D talking avatars with emotional depth and distinctive identity.