Non-Audible Murmur (NAM) speech made a way to convert the silent speech into intelligible speech. However, the weak and un- intelligible nature of NAM signals increases challenges in order to con- vert high-quality speech. While existing architectures effectively handle speech reconstruction, they tend to overlook the nuances of prosody and Intonation. Existing architecture is more focused on the speech recon- struction, which ignores prosody and intonation. To address this, we added a style-based diffusion model and increased layers in seq2seq transformer to improve style consistency and naturalness in the generated speech. This paper proposes NAMTalk, a controllable speech generation, with enhanced intelligibility and expressiveness in NAM2Speech conversion. We integrated a diffusion-based speech synthesis approach to enable a controllable generation of speaking styles and prosody, enhancing the naturalness of the synthesised speech. The generated speech samples were evaluated using subjective (e.g., MOS, NISQA MOS), objective (e.g., MCD, MSD, PESQ, Cosine, STOI), and quantitative (e.g., WER, CER) evaluation metrics. NAMTalk has achieved notable improvement over the baseline model. Between the two proposed models, the second achieved further reductions of 0.0183 in WER and 0.0664 in CER, validating the framework’s effectiveness in generating intelligible speech. The generated samples can be accessed at https://abcdxxxxxx.github.io/NAMtalk-Sample-/ .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

NAMTalk: From Muscle Vibrations to Emotional Speech

  • Kunjan Gajre,
  • Rajnidhi Gupta,
  • Ravindrakumar M. Purohit,
  • Hemant A. Patil

摘要

Non-Audible Murmur (NAM) speech made a way to convert the silent speech into intelligible speech. However, the weak and un- intelligible nature of NAM signals increases challenges in order to con- vert high-quality speech. While existing architectures effectively handle speech reconstruction, they tend to overlook the nuances of prosody and Intonation. Existing architecture is more focused on the speech recon- struction, which ignores prosody and intonation. To address this, we added a style-based diffusion model and increased layers in seq2seq transformer to improve style consistency and naturalness in the generated speech. This paper proposes NAMTalk, a controllable speech generation, with enhanced intelligibility and expressiveness in NAM2Speech conversion. We integrated a diffusion-based speech synthesis approach to enable a controllable generation of speaking styles and prosody, enhancing the naturalness of the synthesised speech. The generated speech samples were evaluated using subjective (e.g., MOS, NISQA MOS), objective (e.g., MCD, MSD, PESQ, Cosine, STOI), and quantitative (e.g., WER, CER) evaluation metrics. NAMTalk has achieved notable improvement over the baseline model. Between the two proposed models, the second achieved further reductions of 0.0183 in WER and 0.0664 in CER, validating the framework’s effectiveness in generating intelligible speech. The generated samples can be accessed at https://abcdxxxxxx.github.io/NAMtalk-Sample-/ .