Improved Alignment for Score Combination of RNN-T and CTC Decoder for Online Decoding

There has been growing interest in utilizing ensembles of CTC and RNN-T models to improve online ASR performance, as diverse model architectures can provide additional information and predictions for both training and decoding. To combine the CTC and RNN-T model predictions, previous works use shallow fusion to combine their frame-level scores during prefix beam search. However, the approach gives inferior results when the scores are not aligned. This is because the RNN-T and CTC models may each emit the same text token after a different number of frames. This misalignment means scores are wrongly combined at frames where one model outputs a text token but another model outputs a blank token. To address this, this paper proposes to align the RNN-T and CTC outputs using a sliding window algorithm to perform text-text matching and avoid wrongly combining the scores at text-blank outputs. On AISHELL-1 and the Singapore National Speech Corpus, our method consistently reduces the character and word error rate from 4.52% to 4.38%, and from 21.29% to 20.03% respectively.