Speech Emotion Recognition (SER) plays a pivotal role in advancing human-machine interactions and mental health monitoring. Despite significant progress in pre-trained models such as wav2vec 2.0, capturing the subtleties of emotion remains challenging due to their emphasis on general acoustic features rather than emotion-specific characteristics. To bridge this gap, we introduce EmoTune, a simple but effective approach that fine-tunes wav2vec 2.0 by integrating an explicit emotion-aware token (i.e., [EMO]) into its architecture. This token, initialized with feature vectors derived from multiple emotion-labeled audio segments, acts as an emotional anchor point, fusing final-layer hidden features to derive comprehensive emotional representations for classification. By freezing the CNN encoder layers during fine-tuning, our method focuses on learning high-level emotional representations, greatly enhancing training efficiency and stability. Furthermore, the integration of prototypical supervised contrastive learning loss significantly boosts the model’s ability to discriminate between various emotional states. Experimental results on the IEMOCAP dataset show that EmoTune outperforms state-of-the-art methods in SER tasks, offering significant potential for future advancements.