Rethinking text-based extractive speech summarization in noisy ASR settings for low-resource language

Text-based speech summarization methods typically rely on Automatic Speech Recognition (ASR) to transcribe spoken input before applying Natural Language Processing (NLP) techniques. However, in low-resource languages like Bengali, ASR systems are often trained on limited and noisy data, leading to transcription errors that degrade summarization performance. Building on a previous study which qualitatively examined how specific ASR error types distort extractive summaries, this paper advances the investigation by providing a quantitative robustness benchmark. In this work, robustness is defined as the ability of a summarization system to maintain its performance in the presence of ASR-induced transcription errors. This study systematically evaluates the robustness of widely-used extractive summarization algorithms—including graph-based (e.g., TextRank), topic-model-based e.g., Latent Dirichlet Allocation (LDA), Latent Semantic Analysis(LSA), and redundancy-aware methods e.g., Maximal Marginal Relevance(MMR), Ranking—in the presence of ASR noise. To assess their resilience, we test each method using four sentence embedding strategies: Term Frequency-Inverse Document Frequency (TF-IDF), FastText, Global Vectors for Word Representation (GloVe), and Cross-Lingual Language Model with RoBERTa (XLM-R). Experiments are conducted on a curated Bengali audiobook corpus with both manual and ASR-generated transcripts. Evaluation using Recall-Oriented Understudy for Gisting Evaluation (ROUGE) metrics reveals that high-quality embeddings like GloVe, when paired with ranking-based methods, maintain competitive performance even with noisy input, achieving a maximum ROUGE-1 score of 0.630. In contrast, low-resource embeddings (e.g., TF-IDF) combined with topic-modeling techniques show severe performance degradation—over 50% drop in ROUGE-2 scores—due to their reliance on lexical overlap and inability to handle disfluencies. Contextual models like XLM-R, while powerful on clean data, exhibit brittleness without fine-tuning. Our findings underscore the importance of selecting summarization strategies that degrade gracefully under imperfect transcriptions, and challenge the common assumption that text is always a reliable intermediate in speech-based NLP pipelines for low-resource settings.