Recent advancements in brain-computer interfaces (BCIs) have leveraged neuroimaging technology and deep learning methods. One promising application is speech BCI, enabling users to control devices through speech imagination. In this study, electroencephalography (EEG) signals of imagined, intended, and spoken speech were collected from 10 native Chinese-speaking subjects. The stimuli used in the experiment consist of monosyllabic Mandarin words, comprising four categories of vowels and four categories of tones. We used an improved EEGNet network to classify the data for the three modalities with accuracy of 91.87%. In addition, we compared the classification performance of EEG signals over different brain regions and frequency bands to investigate the features that make the differences between different speech modalities more pronounced. This study is of great significance to the BCI research of Mandarin speech, and it provides new ideas to enable the model to dynamically adapt to different speech modalities. In addition, this study also helps to deeply explore the neural mechanisms of the speech production process.