Facial Emotion Recognition (FER) is implemented in several applications in computer vision, human computer interaction, as well as emotion recognition. While using deep neural networks for tasks such as FER with ResNet variants, it is necessary to benchmark the best stochastic optimization strategy and a learning rate. This research assesses the influence of cosine annealing, exponential decay, reduce-on-plateau and step decay as learning rate scheduler techniques on training and evaluation of ResNet variants using CK+ dataset. Fixed learnable hyperparameters were used during the optimizations to maintain uniformity by applying Adam, AdamAMSGrad, RMSProp and SGD with momentum optimizers. The results quite clearly stand out that the training, validation and test accuracies can be boosted by applying learning rate scheduler tuning at ResNet50 and ResNet18 architectural levels. Among the various learning schedulers that were used in this experiment, Exponential Decay and Reduce-on-Plateau were observed to be the most efficient and enabled the models to generalize better to any unseen data. The experimental results show the accuracy rate 94.12% with ResNet50 AdamAMSGrad optimization while Exponential decay LR scheduling is used.