Audio-driven singing avatar gesture synthesis represents a key research problem in digital human animation, with wide-ranging applications in media production. Compared with speech, singing exhibits stronger rhythmic patterns, broader pitch dynamics, and richer emotional expressiveness, which introduce new challenges for existing co-speech gesture generation models. To address these challenges, we construct MusicGesture, a full-body singing dataset featuring multi-style synchronized singing videos, corresponding SMPL-X motion parameters, and high-fidelity audio from eight performers. Building upon the EMAGE framework, we further propose SynSinger, a unified model for generating full-body singing motion and lip synchronization directly from raw audio input. By integrating music-aware conditioning pack, lip synchronization refinement, and singing-aware constraints into the generation framework, we enable the synthesis of more natural, synchronized, and expressive full-body motions for digital humans in singing scenarios. Extensive experiments on the MusicGesture dataset demonstrate that SynSinger substantially outperforms baseline models while producing more rhythmically consistent and perceptually natural results. This work not only bridges the gap in full-body pose synthesis for singing scenarios but also introduces a new paradigm for generating rhythm-aware and emotionally expressive digital humans.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

SynSinger: Audio-Driven Singing Avatar Motion Synthesis

  • Yanqi Wang,
  • Zhixin Xu

摘要

Audio-driven singing avatar gesture synthesis represents a key research problem in digital human animation, with wide-ranging applications in media production. Compared with speech, singing exhibits stronger rhythmic patterns, broader pitch dynamics, and richer emotional expressiveness, which introduce new challenges for existing co-speech gesture generation models. To address these challenges, we construct MusicGesture, a full-body singing dataset featuring multi-style synchronized singing videos, corresponding SMPL-X motion parameters, and high-fidelity audio from eight performers. Building upon the EMAGE framework, we further propose SynSinger, a unified model for generating full-body singing motion and lip synchronization directly from raw audio input. By integrating music-aware conditioning pack, lip synchronization refinement, and singing-aware constraints into the generation framework, we enable the synthesis of more natural, synchronized, and expressive full-body motions for digital humans in singing scenarios. Extensive experiments on the MusicGesture dataset demonstrate that SynSinger substantially outperforms baseline models while producing more rhythmically consistent and perceptually natural results. This work not only bridges the gap in full-body pose synthesis for singing scenarios but also introduces a new paradigm for generating rhythm-aware and emotionally expressive digital humans.