Live concert recordings play a crucial role in the music industry but often suffer from complex audio issues not typically encountered in studio productions. These include microphone phase differences, crosstalk, background noise, signal dropouts, clipping, excessive sibilance, vocal distortion, and performance mistakes. Traditional restoration approaches, such as manual editing or re-recording, are labor-inten-sive, costly, and often fail to preserve the authenticity of the live performance. This study presents a generative AI-based system for restoring singing voices in live concert recordings. The proposed framework uses cross-modal feature transformation to repair problematic segments by leveraging musical score information. The system takes as input a short vocal audio clip (4–6 bars), a lyrics file, and the corresponding musical score, and outputs a high-quality, restored vocal segment. The architecture integrates three key components: SOFA, a forced-alignment tool for precise synchronization between vocals and lyrics; DiffSinger, a diffusion-based singing voice synthesis model that extracts phoneme-level features from the score; and VoiceCraft, a text-to-speech model adapted as the core generative module due to its strong acoustic preservation capabilities. A central component of the system is a deep learning-based adapter module that enables cross-modal mapping by transforming musical feature representations into VoiceCraft-compatible text embeddings. Audio generation is carried out through an autoregressive Transformer-based architecture with attention masking and Encodec quantization, ensuring temporal consistency and audio quality. The system effectively retains the original singer’s timbre and the environmental acoustics of the recording, allowing the restored segments to blend seamlessly with the original performance. This research offers a novel, efficient, and high-fidelity solution for vocal restoration in live music recordings, overcoming the limitations of traditional methods and introducing a new technological direction for the music production industry.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Adaptive Singing Voice Enhancement for Live Stages

  • Jia-Lien Hsu,
  • Pei-Wen Chien

摘要

Live concert recordings play a crucial role in the music industry but often suffer from complex audio issues not typically encountered in studio productions. These include microphone phase differences, crosstalk, background noise, signal dropouts, clipping, excessive sibilance, vocal distortion, and performance mistakes. Traditional restoration approaches, such as manual editing or re-recording, are labor-inten-sive, costly, and often fail to preserve the authenticity of the live performance. This study presents a generative AI-based system for restoring singing voices in live concert recordings. The proposed framework uses cross-modal feature transformation to repair problematic segments by leveraging musical score information. The system takes as input a short vocal audio clip (4–6 bars), a lyrics file, and the corresponding musical score, and outputs a high-quality, restored vocal segment. The architecture integrates three key components: SOFA, a forced-alignment tool for precise synchronization between vocals and lyrics; DiffSinger, a diffusion-based singing voice synthesis model that extracts phoneme-level features from the score; and VoiceCraft, a text-to-speech model adapted as the core generative module due to its strong acoustic preservation capabilities. A central component of the system is a deep learning-based adapter module that enables cross-modal mapping by transforming musical feature representations into VoiceCraft-compatible text embeddings. Audio generation is carried out through an autoregressive Transformer-based architecture with attention masking and Encodec quantization, ensuring temporal consistency and audio quality. The system effectively retains the original singer’s timbre and the environmental acoustics of the recording, allowing the restored segments to blend seamlessly with the original performance. This research offers a novel, efficient, and high-fidelity solution for vocal restoration in live music recordings, overcoming the limitations of traditional methods and introducing a new technological direction for the music production industry.