NAMTalk: From Muscle Vibrations to Emotional Speech
摘要
Non-Audible Murmur (NAM) speech made a way to convert the silent speech into intelligible speech. However, the weak and un- intelligible nature of NAM signals increases challenges in order to con- vert high-quality speech. While existing architectures effectively handle speech reconstruction, they tend to overlook the nuances of prosody and Intonation. Existing architecture is more focused on the speech recon- struction, which ignores prosody and intonation. To address this, we added a style-based diffusion model and increased layers in seq2seq transformer to improve style consistency and naturalness in the generated speech. This paper proposes NAMTalk, a controllable speech generation, with enhanced intelligibility and expressiveness in NAM2Speech conversion. We integrated a diffusion-based speech synthesis approach to enable a controllable generation of speaking styles and prosody, enhancing the naturalness of the synthesised speech. The generated speech samples were evaluated using subjective (e.g., MOS, NISQA MOS), objective (e.g., MCD, MSD, PESQ, Cosine, STOI), and quantitative (e.g., WER, CER) evaluation metrics. NAMTalk has achieved notable improvement over the baseline model. Between the two proposed models, the second achieved further reductions of 0.0183 in WER and 0.0664 in CER, validating the framework’s effectiveness in generating intelligible speech. The generated samples can be accessed at https://abcdxxxxxx.github.io/NAMtalk-Sample-/ .