<p>Cascade speech summarization, which typically involves chaining automatic speech recognition (ASR) with text summarization, generates a text summary from speech. However, ASR errors adversely impact performance. To mitigate the propagation of ASR errors, this work explores the use of large language model (LLM) in cascade speech summarization. First, we develop a framework called mapping errors to corrections (METC) for constructing an instruction set, enabling instruction tuning that allows LLM to generate accurate summaries from imperfect transcripts. Second, to transfer the capabilities learned from this set to the summarization task, we investigate three fine-tuning strategies for LLM: two-stage fine-tuning, hybrid fine-tuning, and curriculum-learning fine-tuning. Experimental results demonstrate that the METC instruction set substantially improves the ability of LLM to generate accurate summaries under challenging ASR conditions, with our method outperforming the baseline by 6.93% on the MegaSSum dataset. We further evaluate three fine-tuning strategies on a subset of the TEDsum dataset, in which transcripts generated by the Whisper model are used as input instead of ground-truth text. The results indicate that the curriculum-learning fine-tuning strategy effectively transfers the capabilities acquired from the METC instruction set, achieving a ROUGE-L score for summaries generated from imperfect transcripts that is only one point lower than that for summaries generated from ground-truth transcripts.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Mapping Errors to Corrections: Promoting Self-Correcting LLM in Speech Summarization

  • Xuejian Zhao,
  • Xiaozhe Qi,
  • Hongfu Jia,
  • Zhida Song,
  • Xiang He,
  • Liang He

摘要

Cascade speech summarization, which typically involves chaining automatic speech recognition (ASR) with text summarization, generates a text summary from speech. However, ASR errors adversely impact performance. To mitigate the propagation of ASR errors, this work explores the use of large language model (LLM) in cascade speech summarization. First, we develop a framework called mapping errors to corrections (METC) for constructing an instruction set, enabling instruction tuning that allows LLM to generate accurate summaries from imperfect transcripts. Second, to transfer the capabilities learned from this set to the summarization task, we investigate three fine-tuning strategies for LLM: two-stage fine-tuning, hybrid fine-tuning, and curriculum-learning fine-tuning. Experimental results demonstrate that the METC instruction set substantially improves the ability of LLM to generate accurate summaries under challenging ASR conditions, with our method outperforming the baseline by 6.93% on the MegaSSum dataset. We further evaluate three fine-tuning strategies on a subset of the TEDsum dataset, in which transcripts generated by the Whisper model are used as input instead of ground-truth text. The results indicate that the curriculum-learning fine-tuning strategy effectively transfers the capabilities acquired from the METC instruction set, achieving a ROUGE-L score for summaries generated from imperfect transcripts that is only one point lower than that for summaries generated from ground-truth transcripts.