A key advancement in computer vision is facial expression recognition (FER), which allows computers to decipher complex visual clues to determine human emotions. Its use in security systems, mental health monitoring, and human-computer interaction highlights how important it is for FER systems to have high accuracy and robustness. The Ada-DF architecture is a framework created for FER to tackle the issues of sparse datasets, real-world unpredictability, and inconsistent annotations. This architecture is used as the basis of this paper. Through the testing of the SFEW and AffectNet datasets, we show the possibilities of adaptive learning methods and architectural optimisations. Backbone models, such as ResNet and EfficientNet, outperformed conventional designs, with 60.63% and 65.80% validation accuracies, respectively. Models were able to identify to a validation accuracy of 65.74%. Additionally, a thorough examination of dataset class sizes indicated a need to monitor the accuracy of the models as the class size increases.