<p>The choice of optimization algorithm significantly impacts deep learning model performance, affecting convergence speed, generalization, and training stability. Existing optimizers, such as Adam, AMSGrad, and AdamW, face significant limitations that hinder their effectiveness across diverse deep learning tasks and architectures. These include suboptimal momentum utilization, stalled training due to vanishing learning rates, and compromised generalization with weight decay. To address these critical challenges, we propose Nova, a novel hybrid optimizer that synergistically integrates Nesterov momentum, AMSGrad’s non-decreasing second moment estimate, and decoupled weight decay. Nova introduces a pioneering integration of established techniques, distinguishing it from prior optimizers. Beyond its core components, Nova incorporates adaptive gradient scaling to handle sparse and imbalanced data efficiently and a hybrid learning rate adjustment to reduce hyperparameter sensitivity. This combination enhances training dynamics, stability, and robustness across various deep learning tasks. Experimental results on benchmark datasets, such as CIFAR-10, MNIST, SST-2, and noisy MNIST, demonstrate Nova’s superior performance. For instance, Nova achieves a test accuracy of 90.01% on CIFAR-10, significantly outperforming Adam’s 83.14% and Nadam’s 80.85%. On the SST-2 dataset, Nova achieves a test accuracy of 82.00%, surpassing Adam’s 66.28% and AdamW’s 81.65%. Furthermore, Nova demonstrates robustness to noisy data, achieving a test accuracy of 96.58% on noisy MNIST. These results show that Nova effectively overcomes the challenges of slow convergence, poor generalization, and sensitivity to data characteristics faced by existing optimizers. Nova’s exceptional performance across diverse benchmark datasets suggests its significant potential for broader applications in deep learning. By addressing the critical limitations of existing optimizers through a novel and synergistic combination of techniques, Nova represents a significant advancement in deep learning optimization, offering a powerful and robust solution that enhances training efficiency, model generalization, and stability. The code, additional experimental results, Online Resource <InternalRef RefID="MOESM1">1</InternalRef> and <InternalRef RefID="MOESM2">2</InternalRef> and full text (Online Resource <InternalRef RefID="MOESM3">3</InternalRef>) related to this paper are publicly available at <a href="https://github.com/Aliyar4061/Nova-Optimizer/tree/main">https://github.com/Aliyar4061/Nova-Optimizer/tree/main</a>.</p>

Nova: A Novel Optimizer Integrating Nesterov Momentum, AMSGrad, and Decoupled Weight Decay for Deep Learning

期刊论文
发表日期：

2025年6月10日

查看全文

Ali Zeydi Abdian,
Mohammad Masoud Javidi,
Najme Mansouri

摘要

The choice of optimization algorithm significantly impacts deep learning model performance, affecting convergence speed, generalization, and training stability. Existing optimizers, such as Adam, AMSGrad, and AdamW, face significant limitations that hinder their effectiveness across diverse deep learning tasks and architectures. These include suboptimal momentum utilization, stalled training due to vanishing learning rates, and compromised generalization with weight decay. To address these critical challenges, we propose Nova, a novel hybrid optimizer that synergistically integrates Nesterov momentum, AMSGrad’s non-decreasing second moment estimate, and decoupled weight decay. Nova introduces a pioneering integration of established techniques, distinguishing it from prior optimizers. Beyond its core components, Nova incorporates adaptive gradient scaling to handle sparse and imbalanced data efficiently and a hybrid learning rate adjustment to reduce hyperparameter sensitivity. This combination enhances training dynamics, stability, and robustness across various deep learning tasks. Experimental results on benchmark datasets, such as CIFAR-10, MNIST, SST-2, and noisy MNIST, demonstrate Nova’s superior performance. For instance, Nova achieves a test accuracy of 90.01% on CIFAR-10, significantly outperforming Adam’s 83.14% and Nadam’s 80.85%. On the SST-2 dataset, Nova achieves a test accuracy of 82.00%, surpassing Adam’s 66.28% and AdamW’s 81.65%. Furthermore, Nova demonstrates robustness to noisy data, achieving a test accuracy of 96.58% on noisy MNIST. These results show that Nova effectively overcomes the challenges of slow convergence, poor generalization, and sensitivity to data characteristics faced by existing optimizers. Nova’s exceptional performance across diverse benchmark datasets suggests its significant potential for broader applications in deep learning. By addressing the critical limitations of existing optimizers through a novel and synergistic combination of techniques, Nova represents a significant advancement in deep learning optimization, offering a powerful and robust solution that enhances training efficiency, model generalization, and stability. The code, additional experimental results, Online Resource 1 and 2 and full text (Online Resource 3) related to this paper are publicly available at https://github.com/Aliyar4061/Nova-Optimizer/tree/main.