The miniaturization of EEG devices is essential for the development of consumer-grade brain-computer interface technology. However, low-channel EEG signals exacerbate the inherent disadvantages of low signal-to-noise ratio and low spatial resolution, making the decoding of neural activity even more challenging. To overcome these limitations, we propose an advanced multi-model fusion network that combines temporal-spatial and spectral-spatial features, referred to as the temporal-spectral cross-fusion network (TSCF-Net). This novel architecture consists of two parallel models, i.e., the spectral-spatial model and the temporal-spatial model. In the spectral-spatial model, the one-dimensional EEG time series is first transformed into a two-dimensional time-frequency representation to reveal its intrinsic time-varying characteristics. The time-frequency representation is then extended into the depth dimension to capture the spatial characteristics. On the other hand, the temporal-spatial model directly applies the one-dimensional EEG time series as input, and similarly extends it into the depth dimension to extract spatiotemporal features. To constrain the distribution of these features, a maximum mean discrepancy loss is introduced for feature fusion during the training. Finally, a weighted fusion method is employed to integrate these features. Experimental results based on the BCI Competition IV 2a and IV 2b datasets demonstrate that the TSCF-Net outperforms other baseline methods in low-channel EEG decoding tasks, achieving the highest average accuracy and kappa across all datasets. Additionally, a series of ablation experiments further confirm the effectiveness of the multimodal fusion structure.