Detecting Disfluencies—Overcoming Barriers to Effective Communication
摘要
Disfluencies, such as hesitations, repetitions, and corrections, pose significant challenges in natural language processing (NLP), affecting tasks like speech recognition and language understanding. This study examines various deep learning methods for detecting disfluencies in text and assesses their effectiveness in enhancing NLP systems. We introduce several innovative models, including a binary classification model with Word2Vec embeddings and Bidirectional LSTM for sentence-level tasks, a word-level model utilizing SMOTE for balancing, character encodings, a label encoder for tags, and Bidirectional LSTM, as well as a novel sentence-level model integrating embeddings from both input and tag sentences into a Bidirectional LSTM framework. Evaluations reveal that the Bi-LSTM with Word2Vec embeddings achieves micro and macro F1-scores of 0.75 and 0.73, respectively, for sentence-level tasks, while the Bi-LSTM with multi-embeddings excels with scores of 0.99. For word-level tasks, the Bi-LSTM with character encodings attains micro and macro F1-scores of 0.65 and 0.64. Additionally, fusion models trained on Bengali words show notable performance, with the Bi-LSTM + CRF model achieving F1-scores of 56.8 (word = 59112) and 54.5 (word = 68410). These results underscore the importance of context, linguistic features, and domain specificity in disfluency detection, suggesting pathways for enhancing accuracy and naturalness in NLP systems’ language processing capabilities.