Auditory Attention Decoding aims to identify the attended speech from EEG recordings, often formulated as a match-mismatch classification task. However, current methods suffer from a severe representation imbalance: while speech features are extracted using powerful pre-trained models, EEG encoders remain shallow and under-optimized, limiting overall performance. To address this gap, we propose HERMES—a Heterogeneous Mixture of Experts Based on Segments for EEG encoding. HERMES models EEG signals from three complementary perspectives: local temporal patterns, long-range dependencies, and global attention. Unlike traditional frame-level processing, HERMES operates at the segment level to preserve temporal context and semantic coherence. We further align EEG and speech representations in a shared space via contrastive similarity learning. Experiments on the large-scale SparrKULee dataset demonstrate that HERMES achieves 87.19% accuracy, surpassing the previous state-of-the-art models by over 5%, and exhibiting strong generalization across subjects and stories. Ablation studies further confirm the effectiveness of both the heterogeneous expert design and segment-level routing, each contributing significantly to performance gains. The implementation code will be available on Github: https://github.com/Collin8829/HERMES.git .