Generating talking face videos from audio has gained significant research interest due to its various applications. However, end-to-end approaches that simultaneously reconstruct facial features and synchronize lip movements often struggle to balance these tasks effectively, resulting in low video quality with blurred details and inconsistent identity preservation as well as disconnected facial movements throughout videos. To address these challenges, we propose a novel two-stage framework (LTDAD-Talker) that separates the process into an Audio2Lmk module for generating precise facial landmark sequences and a Lmk2Lip module that creates high-quality facial imagery guided by these landmarks. Within the Lmk2Lip module, we implement an Unet-based generator with Temporal layers interleaved between ResNet layers, enhancing temporal consistency and ensuring smooth facial transitions throughout the video. Additionally, we introduce a Detail-Aware Discriminator (DAD) that comprehensively evaluates the quality of the generated video by analyzing both local features and global facial structure, aiming to significantly enhance sharpness and preserve fine details across the video. Extensive experiments on AVSpeech, MEAD, and HDTF demonstrate that our model outperforms state-of-the-art methods in FID and SSIM, achieving nearly twice the FID improvement over end-to-end approaches. This highlights the advantages of DAD in preserving facial details and the effectiveness of the two-stage approach as well. Additionally, our model achieves the best smoothness scores, validating the Temporal layer’s role in ensuring temporal consistency and smooth motion across frames while maintaining competitive audio-lip synchronization performance in LSE-C and LSE-D metrics.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

LTDAD-Talker: Landmark-Guided Talking Face Generation with Temporal Consistency and Detail-Aware Discriminator

  • Duc Khoan Le,
  • Duc Hao Do,
  • Minh Hoang Pham

摘要

Generating talking face videos from audio has gained significant research interest due to its various applications. However, end-to-end approaches that simultaneously reconstruct facial features and synchronize lip movements often struggle to balance these tasks effectively, resulting in low video quality with blurred details and inconsistent identity preservation as well as disconnected facial movements throughout videos. To address these challenges, we propose a novel two-stage framework (LTDAD-Talker) that separates the process into an Audio2Lmk module for generating precise facial landmark sequences and a Lmk2Lip module that creates high-quality facial imagery guided by these landmarks. Within the Lmk2Lip module, we implement an Unet-based generator with Temporal layers interleaved between ResNet layers, enhancing temporal consistency and ensuring smooth facial transitions throughout the video. Additionally, we introduce a Detail-Aware Discriminator (DAD) that comprehensively evaluates the quality of the generated video by analyzing both local features and global facial structure, aiming to significantly enhance sharpness and preserve fine details across the video. Extensive experiments on AVSpeech, MEAD, and HDTF demonstrate that our model outperforms state-of-the-art methods in FID and SSIM, achieving nearly twice the FID improvement over end-to-end approaches. This highlights the advantages of DAD in preserving facial details and the effectiveness of the two-stage approach as well. Additionally, our model achieves the best smoothness scores, validating the Temporal layer’s role in ensuring temporal consistency and smooth motion across frames while maintaining competitive audio-lip synchronization performance in LSE-C and LSE-D metrics.