Emotion recognition from EEG is vital for affective computing, yet challenged by signal non-stationarity and cross-subject variability, which impede stable representation learning and hinder model generalization. To address these issues, we propose SemGNG-CPC, a novel self-supervised learning framework that integrates Contrastive Predictive Coding (CPC) with a Semantic Growing Neural Gas (SemGNG) network. Its core innovation is the SemGNG module, which adaptively learns a topological manifold that is semantically structured by emotion categories. A built-in cross-subject contrastive regularizer explicitly aligns identical emotional categories across subjects while dispersing dissimilar ones, enhancing feature discrimination and generalization. Extensive evaluations on public SEED and CRED datasets demonstrate the framework’s superiority. SemGNG-CPC achieves state-of-the-art cross-subject recognition accuracy of 84.00% (SEED) and 42.86% (CRED). Topological analysis further verifies that our adaptive approach successfully preserves the intrinsic semantic-topological structure of EEG emotions. This work establishes a new paradigm of semantic-guided adaptive topology learning, paving the way for building robust, interpretable, and generalizable EEG-based emotion recognition systems.