Adapter-Only Bridging of Frozen Speech Encoder and Frozen LLM for ASR

Integrating Large Language Models (LLMs) with speech encoders can improve the performance of Automatic Speech Recognition (ASR) by leveraging enhanced linguistic knowledge. Prior work faces two key limitations. Fine-tuning the LLM may cause forgetting of pre-trained domain knowledge. Methods that freeze both components but use only a single linear projection suffer catastrophic domain-transfer failures. We propose adapter-only bridging that keeps both the speech encoder and the LLM frozen while training lightweight adapters with causal convolutions for temporal modeling (24.6M parameters, 0.44% of total parameters). Our adapters comprise a convolutional downsampler and a multi-layer MLP projection. Trained on 2.9k h of general-domain speech, our approach achieves competitive performance on LibriSpeech and significantly outperforms Whisper-large-v2 on academic domains (26.8% relative WER reduction). Unlike prior work with frozen components that fails on out-of-domain data, our causal convolutional adapters enable robust cross-domain performance while preserving the LLM’s text-trained knowledge for inference-time domain prompting. On four academic domains, domain prompting achieves additional WER reductions in three domains (Engineering 2.10%, Social Sciences 1.30%, Medical 0.73%) and improves domain-specific terminology recognition by 7.2%p F1 and 5.8%p recall. Our approach enables a single frozen model to maintain general-domain accuracy while adapting across domains via prompts alone.