Speech Emotion Recognition (SER) is crucial for enhancing human-machine voice interactions, allowing systems to better interpret the speaker’s emotional state and improve user experience. Self-supervised learning (SSL) models have significantly advanced speech recognition by generating speech representations from large unlabeled corpus, which can then be fine-tuned with smaller labeled datasets for downstream tasks such as voice command detection, automatic transcription, speaker identification, and SER. However, SSL models are typically optimized for general tasks rather than emotion recognition. This poses a challenge, as the limited labeled data in SER can hinder the generalization capabilities of these models, making the choice of SSL architecture and training strategy vital. The most common procedure in Speech Emotion Recognition is performing an average time pooling of the SSL features to train classification models, which often leads to a loss of temporal relationships in the data. In this study, we introduce two novel SSL feature aggregation methods that leverage attention mechanisms to better capture temporal dependencies in speech data. These methods significantly enhance the extraction of relevant information from SSL features, leading to improvements in classification accuracy. Our proposed approach outperforms the standard average time pooling method, achieving up to a 6.3% increase in weighted accuracy (WA) on the IEMOCAP database.