Beyond Universal Transformer: Block Reusing with Adaptor in Transformer for Automatic Speech Recognition

Recently, Transformer-based models have excelled in end-to-end (E2E) automatic speech recognition (ASR), enabling deployment on smart devices. However, their large parameter requirements pose challenges for ASR on edge devices. To address this, we propose a solution that reuses Transformer blocks, optimizing ASR for resource-constrained systems without compromising accuracy. Our novel block-reusing strategy for speech Transformer enhances parameter effectiveness. Additionally, we introduce an adapter module (ADM) that enables a compact and adaptable model by adding just a few trainable parameters to each reusing block. Evaluating on the AISHELL-1 corpus, our approach achieves character error rates (CER) of 8.07%/6.63% with only 7.75M/8.5M parameters, without and with ADM, respectively. Furthermore, we analyze ADM’s impact on the general block-reusing method. Our findings demonstrate the efficacy of block-reusing strategy for speech Transformer (BRST) and the practicality of ADM for efficient ASR on edge devices.