Emotion recognition based on electroencephalography (EEG) is a primary task in affective computing. Meanwhile, multimodal fusion learning improves model performance by leveraging the complementary characteristics of different modalities. While existing studies have successfully identified both emotion categories and the intensity of specific emotions separately, few research has focused on the joint recognition of emotion categories and their intensities. In this paper, we propose a Hierarchical Emotion Transformer (HET) network, which recognizes multiple emotion labels from EEG and eye signals. HET enables the simultaneous prediction of two related emotion labels by employing cross-attention mechanisms with decaying weights and a hierarchical conditional loss function. Furthermore, it can be flexibly extended to cope with any number of multimodal inputs by utilizing attention-based fusion structures. Experimental results on a public dataset with seven emotion categories and corresponding intensity scores show that our model surpasses state-of-the-art performance in both emotion intensity recognition and overall classification accuracy.