Understanding human emotion is a complex and nuanced task; Russell’s circumplex model of affect provides a theoretical foundation for this by organizing emotions into a circular structure. Using a data-driven approach, our study aims to understand how emotions are organized in the latent spaces of Transformer-based models and if this organization aligns with Russell’s psychological-based organization. We applied Transformer-based models for feature extraction from text and audio data, followed by dimensionality reduction to uncover emotional manifolds. By calculating the cosine similarity between the centers of Russell’s affective states in high-dimensional representation and evaluating permutations, we sought to reproduce the circular order of emotions. Our findings reveal that while biased unimodal datasets partially align with Russell’s model, representative data shows that the multimodal approach closely replicates the structure. Our approach’s results, to some extent, decode and validate Russell’s Model of Affect, highlighting the advantages of modality fusion in emotion research.