Scholarlink

<p>Gradient descent-based optimization algorithms are widely used in deep learning, where adaptive optimizers dynamically adjust parameter updates to improve training efficiency. However, most existing adaptive optimizers mainly rely on historical gradients while overlooking short-term gradient fluctuations. Although methods such as diffGrad and AngularGrad attempt to address this issue through gradient-difference or angle-based correction mechanisms, their performance improvements are often task-dependent, and some approaches introduce additional computational overhead due to complex vector operations. To address these limitations, this paper proposes a lightweight short-term gradient-aware mechanism based on gradient sign consistency, which improves optimization stability while maintaining low computational complexity. Based on this idea, two Adam-based optimizers, sigSignGrad and tanhSignGrad, are developed by incorporating adaptive friction coefficients constructed through Sigmoid and Tanh mappings, respectively. In addition, the proposed friction coefficient can be flexibly integrated into the Adam optimizer family as a lightweight plug-in module. Extensive experiments are conducted across multiple computer vision and natural language processing tasks, including image classification, object detection, semantic segmentation, machine translation, language modeling, and language understanding benchmarks. To evaluate the robustness and generalization capability of the proposed methods, multiple datasets and model architectures are considered within the same task settings, covering both training-from-scratch and pre-trained fine-tuning scenarios. Experimental results show that sigSignGrad achieves the best overall performance on multiple visual tasks, including all evaluated Vision Transformer benchmarks, Pascal VOC object detection, and semantic segmentation datasets. In natural language processing tasks, tanhSignGrad achieves the best BLEU scores on the IWSLT14 De-En translation task and consistently outperforms existing optimizers in language modeling experiments under both pre-trained and training-from-scratch settings. Furthermore, the proposed methods maintain computational efficiency comparable to diffGrad while providing more stable optimization behavior across diverse architectures and datasets.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

The Gradient Descent Based Adaptive Optimizer Algorithm in Deep Learning

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月22日

查看全文
  • Rongjian Chen,
  • Ruyan Xiong,
  • Xiaodong Yang

摘要

Gradient descent-based optimization algorithms are widely used in deep learning, where adaptive optimizers dynamically adjust parameter updates to improve training efficiency. However, most existing adaptive optimizers mainly rely on historical gradients while overlooking short-term gradient fluctuations. Although methods such as diffGrad and AngularGrad attempt to address this issue through gradient-difference or angle-based correction mechanisms, their performance improvements are often task-dependent, and some approaches introduce additional computational overhead due to complex vector operations. To address these limitations, this paper proposes a lightweight short-term gradient-aware mechanism based on gradient sign consistency, which improves optimization stability while maintaining low computational complexity. Based on this idea, two Adam-based optimizers, sigSignGrad and tanhSignGrad, are developed by incorporating adaptive friction coefficients constructed through Sigmoid and Tanh mappings, respectively. In addition, the proposed friction coefficient can be flexibly integrated into the Adam optimizer family as a lightweight plug-in module. Extensive experiments are conducted across multiple computer vision and natural language processing tasks, including image classification, object detection, semantic segmentation, machine translation, language modeling, and language understanding benchmarks. To evaluate the robustness and generalization capability of the proposed methods, multiple datasets and model architectures are considered within the same task settings, covering both training-from-scratch and pre-trained fine-tuning scenarios. Experimental results show that sigSignGrad achieves the best overall performance on multiple visual tasks, including all evaluated Vision Transformer benchmarks, Pascal VOC object detection, and semantic segmentation datasets. In natural language processing tasks, tanhSignGrad achieves the best BLEU scores on the IWSLT14 De-En translation task and consistently outperforms existing optimizers in language modeling experiments under both pre-trained and training-from-scratch settings. Furthermore, the proposed methods maintain computational efficiency comparable to diffGrad while providing more stable optimization behavior across diverse architectures and datasets.