Recently, Representation Finetuning (ReFT) based methods operate by strategically manipulating a small portion of a large language model’s representation to guide its behavior to more effectively solve downstream tasks. However, current ReFT-based methods like LoReFT and DiReFT have not been well examined for their effectiveness on Speech Processing Tasks. In this paper, we introduces a representation-efficient transfer learning framework for pre-trained speech models, evaluating LoReFT and DiReFT on both automatic speech recognition (ASR) and speech-to-text translation (STT). Unlike conventional fine-tuning that updates all model weights, our approach selectively modifies hidden representations via targeted interventions, reducing parameter updates by orders of magnitude while preserving task performance. Experiments demonstrate that LoReFT-based method achieves near full-fine-tuning performance on speech-to-text translation (BLEU: 38.08 vs. 40.00) using only 0.0007% trainable parameters, and surpasses full fine-tuning in speech recognition (WER: 7.16% vs. 8.63%). DiReFT further reduces memory costs through simplified interventions, albeit with marginal accuracy trade-offs. Our code is available at ( https://anonymous.4open.science/r/RETL-ASR-STT ).

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Exploring Representation-Efficient Transfer Learning Approaches for Speech Recognition and Translation Using Pre-trained Speech Models

  • Wenhao Wang,
  • Yonghe Wang,
  • Nan Chen,
  • Feilong Bao

摘要

Recently, Representation Finetuning (ReFT) based methods operate by strategically manipulating a small portion of a large language model’s representation to guide its behavior to more effectively solve downstream tasks. However, current ReFT-based methods like LoReFT and DiReFT have not been well examined for their effectiveness on Speech Processing Tasks. In this paper, we introduces a representation-efficient transfer learning framework for pre-trained speech models, evaluating LoReFT and DiReFT on both automatic speech recognition (ASR) and speech-to-text translation (STT). Unlike conventional fine-tuning that updates all model weights, our approach selectively modifies hidden representations via targeted interventions, reducing parameter updates by orders of magnitude while preserving task performance. Experiments demonstrate that LoReFT-based method achieves near full-fine-tuning performance on speech-to-text translation (BLEU: 38.08 vs. 40.00) using only 0.0007% trainable parameters, and surpasses full fine-tuning in speech recognition (WER: 7.16% vs. 8.63%). DiReFT further reduces memory costs through simplified interventions, albeit with marginal accuracy trade-offs. Our code is available at ( https://anonymous.4open.science/r/RETL-ASR-STT ).