The impact of sleep states on emotion has garnered significant attention in both psychology and neuroscience. This paper proposes a Transformer-based model to leverage multi-modal effects from temporal, spectral, and spatial views of EEG data: Multi-View Fusion Transformer (MVFT). The advantage of this method lies in its ability to effectively enhance model performance with only a single modality. Meanwhile, we collect a new dataset with 20 subjects participating emotion experiments under sleep deprivation, sleep recovery, and normal sleep. The experimental results indicate that, compared to the uni-modal baseline models, our approach significantly improves the recognition performance which demonstrates that our solution can more fully utilize the effective information within a single modality. Furthermore, the accuracy under the sleep deprivation conditions is generally lower than those under normal sleep and sleep recovery conditions, highlighting the significant impact of sleep conditions on emotional states, which is consistent with previous research findings. In addition, we also study the contribution of EEG data from different views to the emotion recognition task. The results show that the contribution of the temporal and spectral view is significantly greater than that of the spatial view.