Attention-Based Deep Convolutional Network for Speech Recognition Under Multi-scene Noise Environment

One goal of Automatic Speech Recognition (ASR) is to convert the command of human speech into computer-readable input, but noise interference is an important yet challenging problem. Capturing speech context, deep neural networks have demonstrated to be superior in learning networks for identifying specific command words. Existing deep neural networks generally rely on the two-layer structure, different layer is used to identify the noisy environment and speech respectively, which makes the model large and complex. In addition, their performance generally drops dramatically in unknown noisy environments, which restricts the generalization of the method. In this paper, we propose a novel deep framework, named Adaptive-Attention and Joint Supervision (AJS) to circumvent the above challenge. Specifically, we use the spectrogram as the input. Adaptive attention is employed to refine the features from the noise environment and get rid of the limitation of the noisy scene. Furthermore, a combination of coarse-to-fine losses are adopted to process difficult words step by step. Extensive experiments on four public datasets demonstrate the robustness of our method to various noise environments and its superiority for ASR in terms of accuracy. Codes are available at: https://github.com/zhishulin/bajs .