Learning Rate Schedules and Optimizers, A Game Changer for Deep Neural Networks

The Learning Rate (LR) in the Deep Neural Network (DNN) training process determines whether and how quickly the training process may converge. LRs are most effective when the direction and timing of network parameter updates are specified. Note that the optimizers provide the LR update trajectory. The complex interactions between LR and optimizers create a multidimensional environment that calls for systematic investigation. Most existing literature studied the impact of these factors independently, thereby providing limited connecting information between the two. This paper, therefore, offers investigative insights into the complex connections between LR and optimizers through a comprehensive collection of experiments using three datasets in the domain of churn analysis in the telecommunication industry. Nine different LR techniques were employed in the first part of the experiment with optimizers, while seven optimizers were analyzed and compared in the second part. Consequently, the experimental results provide research enthusiasts with the understanding inherent in hyperparameters combination to obtain the best convergence rates and generalization performance as mechanics driving DNN training.