Scholarlink

<p>Within the current sphere of deep learning research, despite the extensive application of optimization algorithms such as Stochastic Gradient Descent (SGD) and Adaptive Moment Estimation (Adam), there remains a pronounced inadequacy in their capability to address fluctuations in learning efficiency, meet the demands of complex models, and tackle non-convex optimization issues. These challenges primarily arise from the algorithms’ limitations in handling complex data structures and models, for instance, difficulties in selecting an appropriate learning rate, avoiding local optima, and navigating through high-dimensional spaces. To address these issues, this paper introduces a novel optimization algorithm named DWMGrad. This algorithm, building on the foundations of traditional methods, incorporates a dynamic guidance mechanism reliant on historical data to dynamically update momentum and learning rates. This allows the optimizer to flexibly adjust its reliance on historical information, adapting to various training scenarios. This strategy not only enables the optimizer to better adapt to changing environments and task complexities but also, as validated through extensive experimentation, demonstrates DWMGrad’s ability to achieve faster convergence rates and higher accuracies under a multitude of scenarios. Specifically, our experiments include assessments on classical tasks in computer vision, natural language processing, and audio processing, wherein DWMGrad has demonstrated exceptional classification accuracy and optimization efficiency in CIFAR-10, CIFAR-100 and ImageNet image classification tasks, NLP and graph classification tasks based on the Roberta model. In audio classification tasks, this method has also proven its performance to be comparable to that of classical optimization methods. Furthermore, our algorithm was tested on the Rosenbrock function to further affirm its universality and robustness. Our research not only validates the effectiveness of DWMGrad in traditional deep learning tasks but also showcases its potential in addressing complex optimization challenges. Ultimately, theoretical proofs confirming the convergence properties of the DWMGrad optimizer are provided, further corroborating the effectiveness and feasibility of our approach in surmounting the challenges of deep learning optimization. The source code of this study is publicly available at <a href="https://github.com/CCNUZFW/DWMGrad">https://github.com/CCNUZFW/DWMGrad</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

DWMGrad: an innovative neural network optimization approach using dynamic window data for adaptive updating of momentum and learning rate

  • 期刊论文

  • 发表日期：

    2025年10月15日

查看全文
  • Zhifeng Wang,
  • Longlong Li,
  • Chunyan Zeng

摘要

Within the current sphere of deep learning research, despite the extensive application of optimization algorithms such as Stochastic Gradient Descent (SGD) and Adaptive Moment Estimation (Adam), there remains a pronounced inadequacy in their capability to address fluctuations in learning efficiency, meet the demands of complex models, and tackle non-convex optimization issues. These challenges primarily arise from the algorithms’ limitations in handling complex data structures and models, for instance, difficulties in selecting an appropriate learning rate, avoiding local optima, and navigating through high-dimensional spaces. To address these issues, this paper introduces a novel optimization algorithm named DWMGrad. This algorithm, building on the foundations of traditional methods, incorporates a dynamic guidance mechanism reliant on historical data to dynamically update momentum and learning rates. This allows the optimizer to flexibly adjust its reliance on historical information, adapting to various training scenarios. This strategy not only enables the optimizer to better adapt to changing environments and task complexities but also, as validated through extensive experimentation, demonstrates DWMGrad’s ability to achieve faster convergence rates and higher accuracies under a multitude of scenarios. Specifically, our experiments include assessments on classical tasks in computer vision, natural language processing, and audio processing, wherein DWMGrad has demonstrated exceptional classification accuracy and optimization efficiency in CIFAR-10, CIFAR-100 and ImageNet image classification tasks, NLP and graph classification tasks based on the Roberta model. In audio classification tasks, this method has also proven its performance to be comparable to that of classical optimization methods. Furthermore, our algorithm was tested on the Rosenbrock function to further affirm its universality and robustness. Our research not only validates the effectiveness of DWMGrad in traditional deep learning tasks but also showcases its potential in addressing complex optimization challenges. Ultimately, theoretical proofs confirming the convergence properties of the DWMGrad optimizer are provided, further corroborating the effectiveness and feasibility of our approach in surmounting the challenges of deep learning optimization. The source code of this study is publicly available at https://github.com/CCNUZFW/DWMGrad.