Music-driven dance generation technology converts audio signals into expressive body movements and is widely used in virtual content creation and intelligent performing arts. However, this task faces challenges in cross-modal alignment, temporal consistency, and visual fidelity, especially in accurately capturing the rhythm of music and mapping it to complex movements. To address these issues, we propose DreamDancer, a cross-modal framework that integrates motion generation and high-quality video synthesis. First, we design a Music-to-Pose module that combines Graph Convolutional Networks with temporal attention to generate rhythm-consistent and stylistically coherent dance sequences. Subsequently, we employ a diffusion-based video synthesis pipeline that is conditioned on both skeletal sequences and a reference image, enhancing the realism and visual quality of generated movements. To mitigate facial identity distortion, we incorporate a learnable face enhancement module based on ArcFace, which applies adaptive feature fusion to improve facial consistency across video frames. Additionally, to increase the model’s ability to generalize across different dance styles, we build a multi-ethnic, multi-modal dance dataset, which enriches cultural and stylistic diversity. Experimental results and user studies show that DreamDancer outperforms existing methods in dance quality, music synchronization, and facial consistency, demonstrating strong practical potential.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DreamDancer: Music-Driven Dance Video Intelligent Generation

  • Dongjin Huang,
  • Jiyu Qian,
  • Yufei Liu,
  • Wenyun Tu,
  • Yichuan Liu

摘要

Music-driven dance generation technology converts audio signals into expressive body movements and is widely used in virtual content creation and intelligent performing arts. However, this task faces challenges in cross-modal alignment, temporal consistency, and visual fidelity, especially in accurately capturing the rhythm of music and mapping it to complex movements. To address these issues, we propose DreamDancer, a cross-modal framework that integrates motion generation and high-quality video synthesis. First, we design a Music-to-Pose module that combines Graph Convolutional Networks with temporal attention to generate rhythm-consistent and stylistically coherent dance sequences. Subsequently, we employ a diffusion-based video synthesis pipeline that is conditioned on both skeletal sequences and a reference image, enhancing the realism and visual quality of generated movements. To mitigate facial identity distortion, we incorporate a learnable face enhancement module based on ArcFace, which applies adaptive feature fusion to improve facial consistency across video frames. Additionally, to increase the model’s ability to generalize across different dance styles, we build a multi-ethnic, multi-modal dance dataset, which enriches cultural and stylistic diversity. Experimental results and user studies show that DreamDancer outperforms existing methods in dance quality, music synchronization, and facial consistency, demonstrating strong practical potential.