This paper describes a novel deep learning approach to recognize emotions associated to facial expressions in the use of CNN and RNN classifiers. The system focuses on improving the identification of emotion with the help of CNNs to extract spatial features of faces and RNNs to capture temporal dynamics of faces’ expressions. This was done by conducting experiments on datasets commonly used like FER2013 and CK+, the system performed satisfactorily in analyzing emotions such as happiness, sadness, anger, and surprise. However, some difficulties are still allows for, for instance, variability across cultures, occlusions, and the computation intensiveness of deep learning models. Also, some ethical issues such as bias and facilitation of model interpretability were also pointed out. The future work shall be dedicated to the multimodal methods’ integration, increasing the real-time capabilities, and the development of the fair-aware machine learning algorithms. This paper also reveals the strengths and weakness of using deep learning in emotion recognition to encourage further research in the field.