Typically, facial emotion recognition (FER) is approached by considering emotions as categorical labels, ignoring the severity of misclassifications - for instance, confusing “disgust” with “angry” is treated equally to confusing either with “surprise”. In this study, we propose a novel multi-task learning framework that optimizes two objectives: a classification head to maximize recognition accuracy, and an ordinal regression head to reduce misclassification severity. For the classification branch, we evaluate five deep architectures (VGG16, ResNet-18, Inception-v3, MobileNet, and ShuffleNet). In the ordinal branch, emotion labels are reordered along the valence or arousal dimensions of Russell’s circumplex model of affect and processed using the Consistent Rank Logits method. The model is trained with a combined loss function that integrates cross-entropy and weighted ordinal binary cross-entropy terms. Experimental results demonstrate that our approach can yield a 1–5% increase in accuracy while substantially reducing the severity of misclassifications compared to baseline methods.