Facial emotion recognition (FER) is a challenging task that involves analyzing emotions based on attributes entangled with the face in an image. The FER model inherently possesses age bias because age-related features such as wrinkles and skin texture can create confusion in understanding the emotions of the elderly and decrease accuracy. This paper proposes a FER method that mitigates age bias by applying adversarial regularization to the value vector of self-attention, which is related to age attributes, to minimize information loss. The proposed method focuses on removing biases without affecting the landmarks that make up facial emotions, as key and query vectors in the attention are associated with facial landmarks while value vector is associated with textural features of the image. An age discriminator interrupts age information in value vectors and restores information through the emotion classifier to minimize excessive loss. Experimental results with two FER datasets, RAF-DB and FACES, show that the proposed method yields superior accuracy while reducing bias compared to the state-of-the-art methods.