Electroencephalogram (EEG)-based emotion recognition is an emerging research area in brain-computer interface (BCI) providing a direct window into one’s cognitive states. Recent studies employ deep learning models such as a convolutional neural network (CNN), a long short-term memory (LSTM), and the Transformer owing to their high performances achieved for EEG-based emotion recognition. Despite their significant research outcomes, individual networks have their respective limitations in their modeling capabilities. To learn complementary feature representations, we cascade global and sequential temporal representations with local context modeling by unifying CNN, Transformer and LSTM into one framework. To verify the effectiveness of our proposed model, we conducted extensive comparative experiments on two popular benchmark datasets for EEG-based emotion recognition, i.e., SEED-IV, and DEAP, in which we bring further improvements over the recent state-of-the-art models. Our code is publicly available at: https://github.com/affctivai/ConTL .