Understanding brainwave dynamics is critical for progress in neuroscience, affective computing, and mental health technologies. However, existing approaches often rely on unimodal processing of EEG signals—either modeling raw temporal sequences or extracting handcrafted spectral features—limiting their ability to generalize across subjects and datasets. Furthermore, few models explicitly learn modality-invariant representations that disentangle emotional information from subject-specific or session-dependent noise. This work introduces Contrastive Time-Frequency Pretraining (CTFP), a deep learning framework that integrates both time-domain EEG signals and frequency-domain Mel-Frequency Cepstral Coefficients (MFCC) representations. Inspired by contrastive learning paradigms such as Contrastive Language-Image Pre-Training (CLIP), the proposed CTFP model aligns temporal and spectral views of the same EEG signal in a shared embedding space, encouraging the model to capture emotion-relevant patterns while being robust to nuisance factors. Extensive experiments on three benchmark datasets—SEED, Brainwave EEG, and WESAD—demonstrate that CTFP achieves state-of-the-art (SOTA) performance in emotion recognition, highlighting its effectiveness and generalizability across diverse recording conditions.