Sub-layer feature fusion applied to transformer model for automatic speech recognition

The Transformer model, an attention-based encoder-decoder network, is popular in the field of automatic speech recognition (ASR) and many researchers have focused on it to improve its model performance. In particular, many of them have focused on feature fusion schemes, which can extract more representative features. However, they treat the encoder layer or decoder layer as a whole and neglect the output features in the sub-layers. Moreover, the importance of sub-layers in the Transformer model is different, so fusing the features in unimportant sub-layers tends to have a negative impact on the model performance. In order to improve the performance of Transformer in ASR field, a sub-layer feature fusion (SLFF) scheme is proposed to fuse the features in important sub-layers. In this case, the fused feature tends to be more representative than the features in the original model. In addition, the SLFF-Transformer is proposed with the integration of the SLFF module and the conventional Transformer model. Finally, empirical evaluations have been conducted on two widely used Chinese Mandarin datasets, which are Aishell-1 and HKUST, to validate the effectiveness of the proposed model. Experimental results show that the proposed model outperforms the state-of-the-art approaches.