During the process of speech perception, the listener’s electroencephalographic(EEG) signals is synchronized with acoustic features such as the speech envelope. This neural tracking mechanism can be used to decode the speech information form the EEG signals, much work has been devoted to investigating. In terms of the limited fitting ability of linear models, many deep learning-based models have been proposed in this field. Recently, Transformer-based models have showed significant potential in the EEG tasks. The Auditory EEG Decoding Challenge 2023 released two tasks to associate a person’s EEG signals with the speech they are listening to, namely match-mismatch and regression. In this paper, two Transformer-based models are proposed for the two auditory downstream tasks. The convolution layer and self-attention mechanism are utilized simultaneously to extract both local features and global dependencies. For the match-mismatch task, the Transformer-Dilated Convolution Network is proposed to classify the speech segments that match the EEG segment. Meanwhile, we design the Transformer-Conformer Network to reconstruct the speech envelope for the regression task. Results show that our proposed models outperform the baseline on both tasks. In addition, the Transformer-Conformer Network is superior in performance comparing with all the challenge teams on the regression track.