Facial expression recognition in the wild is a common but challenging task influenced by different disturbances, which can be divided into non-face-related disturbances and face-related disturbances. Among them, face-related disturbances are intrinsical issue in recognizing a facial expression. However, existed methods based on solving these face-related disturbances ignore the influence of expression intensity on the recognition. In addition, treating the neutral category, which has prominent semantic difference from other categories, equally with other categories will lower the discriminative performance of the networks. This paper introduces the concept of amplitude and phase for facial expression recognition, which reflects the intensity and the category of expression, respectively, and an amplitude phase separation method is proposed to eliminate the disturbances of expression intensity. The amplitude branch distinguishes the facial samples with and without facial expressions. The phase branch utilizes the values fed from the amplitude to separate the amplitude disturbances. The phase centers are set for facial images with different expressions to achieve intra-class compact. The proposed method achieves superior results on the existing facial expression recognition datasets.