Exploring emotional complexity and ambiguity in conversations to identify speaker intent has emerged as a significant trend. Existing works on Emotion Recognition in Conversation (ERC) typically employ one-hot labels as ground truth, ignoring the complexity of emotion. However, in real life, the perception and labeling of complex emotions is a challenging task. To address this challenge, this paper proposes a novel “Emotional Atmosphere Soft Label” generation strategy. This strategy utilizes the emotional atmosphere present in conversations to reveal potential emotional components that may be neglected by hard or soft labels. The emotional atmosphere soft labels are grounded in psychological theories rather than being generated automatically by models. Furthermore, this paper introduces the concept of emotional atmosphere vector, aiming to quantify emotional atmosphere categories and their intensity. Specifically, considering the characteristics of emotional dynamics and the perceptual differences among annotators, this research designs three emotional atmosphere vectors: the Local Emotional Atmosphere Vector, the Annotator-Averaged Emotional Atmosphere Vector, and the Annotator-Self-Adjusted Emotional Atmosphere Vector. Subsequently, by employing the proposed Emotion Atmosphere Adjustment Algorithm, emotional atmosphere vectors are integrated into the labels for each utterance, thereby obtaining three types of soft labels containing potential emotional atmosphere information. The experiments demonstrate that the emotional atmosphere soft labels are compatible with most existing ERC models and datasets. For the IEMOCAP and MELD datasets, emotional atmosphere soft labels assist the ERC models achieve a 0.3–5.8 \(\%\) weighted F1 score improvement. Moreover, emotional atmosphere soft labels significantly alleviate the issue of model overfitting.