Scholarlink

<p>Learning rate is the most important parameter for neural network performance optimization, and its selection and theoretical exploration have always been an active research field. Although the learning rate scheduler has been widely observed to improve the performance of model training, the current works do not consider the characteristics of different stages of training and are mostly incompatible with adaptive optimizers. In this work, we construct a general learning rate scheduler based on the different stages of the training process "warm-up - maintain large learning rate - decay", which can meet the changes of learning rate in different training stages while reducing the dependence on hyperparameters. In addition, we theoretically demonstrate the convergence of the proposed general learning rate scheduler on the adaptive optimizers. Compared with the mainstream learning rate scheduler on the current common data sets, we comprehensively evaluate the advanced and robust performance of the proposed learning rate scheduler through a large number of experiments.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A general learning rate improvement strategy for deep neural networks training

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年1月31日

查看全文
  • Tingting Wu,
  • Qingwei Dong,
  • Wei Yu,
  • Zheng Li,
  • Manxue Guo

摘要

Learning rate is the most important parameter for neural network performance optimization, and its selection and theoretical exploration have always been an active research field. Although the learning rate scheduler has been widely observed to improve the performance of model training, the current works do not consider the characteristics of different stages of training and are mostly incompatible with adaptive optimizers. In this work, we construct a general learning rate scheduler based on the different stages of the training process "warm-up - maintain large learning rate - decay", which can meet the changes of learning rate in different training stages while reducing the dependence on hyperparameters. In addition, we theoretically demonstrate the convergence of the proposed general learning rate scheduler on the adaptive optimizers. Compared with the mainstream learning rate scheduler on the current common data sets, we comprehensively evaluate the advanced and robust performance of the proposed learning rate scheduler through a large number of experiments.