Scholarlink

<p>Fluency evaluation is an important criterion in determining the proficiency level of non-native speakers. Traditional speech fluency models often employ either audio or textual features and lead to poor recognition efficiency. This study develops a multimodal approach that uses both textual and audio features to perform fluency evaluation. The Wav2Vec 2.0 and ALBERT are used for preliminary feature extraction by utilizing speech data from the Avalinguo and Speechocean762 datasets. The Dual Action Shallow and Deep Interaction (DASDI) module containing shallow and deep feature extractors efficiently fuses the features and facilitates cross-modal alignment. The shallow extractor employs text-guided and audio-guided cross-attention to capture semantic and prosodic patterns, whereas temporal attention in the shallow extractor derives evolving disfluency dynamics. A Structured Adaptive Learning (SAL) module selectively emphasizes or suppresses important features to boost cross-modal alignment and minimize computational burden. The deep extractor employs a Transformer with 1D Global Criss-Cross Attention (1D-GCC) to derive global contextual dependencies. Finally, a Bidirectional Long Short-Term Memory (Bi-LSTM) predictor is used to determine fluency levels from multimodal representations. The proposed approach is compared against existing methods. The model attained 98.39% accuracy, 98.28% recall, 98.45% precision, 98.36% F1-score, mean squared error (MSE), 0.321 word error rate (WER), and 0.0231 character error rate (CER). The method outperformed existing methods, including MM-ASR, E2E, CNN, DLNLP-RSD, LSTM Transformer, and others. The results of the statistical analysis showed statistically significant improvements (<i>p</i> &lt; 0.05) and indicate the effectiveness of a multi-modal approach in speech fluency assessment.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A Novel Deep Learning Approach for Fluency Level Prediction in Non-native Speakers Using ASR and Bi-LSTM

  • 期刊论文

  • 发表日期：

    2025年12月4日

查看全文
  • Huifang Guo

摘要

Fluency evaluation is an important criterion in determining the proficiency level of non-native speakers. Traditional speech fluency models often employ either audio or textual features and lead to poor recognition efficiency. This study develops a multimodal approach that uses both textual and audio features to perform fluency evaluation. The Wav2Vec 2.0 and ALBERT are used for preliminary feature extraction by utilizing speech data from the Avalinguo and Speechocean762 datasets. The Dual Action Shallow and Deep Interaction (DASDI) module containing shallow and deep feature extractors efficiently fuses the features and facilitates cross-modal alignment. The shallow extractor employs text-guided and audio-guided cross-attention to capture semantic and prosodic patterns, whereas temporal attention in the shallow extractor derives evolving disfluency dynamics. A Structured Adaptive Learning (SAL) module selectively emphasizes or suppresses important features to boost cross-modal alignment and minimize computational burden. The deep extractor employs a Transformer with 1D Global Criss-Cross Attention (1D-GCC) to derive global contextual dependencies. Finally, a Bidirectional Long Short-Term Memory (Bi-LSTM) predictor is used to determine fluency levels from multimodal representations. The proposed approach is compared against existing methods. The model attained 98.39% accuracy, 98.28% recall, 98.45% precision, 98.36% F1-score, mean squared error (MSE), 0.321 word error rate (WER), and 0.0231 character error rate (CER). The method outperformed existing methods, including MM-ASR, E2E, CNN, DLNLP-RSD, LSTM Transformer, and others. The results of the statistical analysis showed statistically significant improvements (p < 0.05) and indicate the effectiveness of a multi-modal approach in speech fluency assessment.