Facial expression recognition (FER) is a challenging task of computer vision and artificial intelligence. In this paper, we propose a new end-to-end framework for FER, which combines the capsule network and deep metric learning. First, the capsule neural network (CapsNN) was designed to extract feature embeddings from the facial images with varied poses and emotions. Then, a new emotional metric loss function was applied to suppress the intraclass feature differences and enlarge the interclass variations. The proposed method shows a satisfactory performance of capturing latent geometry information from facial images, and its emotional metric loss function also categorizes the seven expressions accurately. Comparative experiments and visualization results on three widely used datasets (i.e., MMI, Oulu-CASIA, and RAF-DB) indicate that the proposed method outperforms some state-of-the-art methods.