Accurate and Early Detection of Dysarthria Using a Multimodal Deep Learning Methodology

Dysarthria is a speech disorder that is brought on by neurological problems. This alters the intelligibility of speech and muscles participating in speech. So early detection is necessary with regular observation. Traditional detection takes time and resources. This process is inefficient. In this paper, we designed a multimodal architecture using the deep learning method. This multimodal can classify dyserthic and non-dyserthic with high precision. It can capture the complex relation between intelligibility and speech articulation. The acoustic data is processed using the Short-Time Fourier Transform (STFT) (D. Griffin and Jae Lim, “Signal estimation from modified short-time Fourier transform,” in IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 32, no. 2, pp. 236–243, 1984.) to generate spectrograms. These spectrograms are passed through a custom-designed Convolutional Neural Network (CNN) (Alzubaidi et al. in J Big Data 8:53, 2021) model for feature extraction. It helps to detect the tone and coordination of muscles used for speech, like lips, tongue, and voice box. Simultaneously, metadata features such as Whisper-based transcription similarity scores and retry attributes are processed through a separate pipeline to detect speech intelligibility and coordination between the brain and muscles used for speech. These two pipelines concatenate and make a dense layer. This multimodal model is trained and evaluated on a publicly available dysarthric speech dataset. It attained an average validation accuracy of 98.20% on the TORGO dataset and 99.40% and a peak fold accuracy of 98.96% on the TORGO dataset and 99.83% on the UA Speech dataset. The confusion matrix shows strong generalization with nearly perfect precision and a high F1 score of 98.97% on the TORGO dataset and 99.84% on the UA Speech dataset, which is more important than accuracy in the medical field.