Scholarlink

<p>Overfitting and slow convergence are common problems when using the gradient descent method in neural network training. We use regularization and momentum terms, respectively, to reduce these negative phenomena. In this paper, we consider the convergence of the gradient descent method with smoothing L<sub>0</sub> regularization and an adaptive momentum term. The objective function for normal L<sub>0</sub> regularization is the sum of a function that is not convex, smooth, or Lipschitz. This makes the error function and the norm of the gradient oscillate. This impediment prevents neural networks from achieving optimal measurement rates for application verification. However, we can address the deficiency of the normal L<sub>0</sub> regularization term by using the smoothing approximation techniques. This paper presents the results of weak convergence for smooth L<sub>0</sub> regularization using the adaptive momentum method. Furthermore, we have proven the strong convergence results of the theorems. Simulations based on three learning problems—parity problems, function approximation problems, and classification tasks—illustrate the viability of the suggested approach under these circumstances. The suggested approach assumes the momentum coefficient, regularization parameter, and learning rate to be constants. We selected these problems because their unique error surfaces provide an appropriate setting for evaluating the efficacy of the suggested approach. Simulation examples demonstrate the superiority of the suggested algorithm and bolster the theoretical analysis.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Convergence analysis of gradient descent based on smoothing L0 regularization and momentum terms

  • 期刊论文

  • 发表日期：

    2025年4月28日

查看全文
  • Khidir Shaib Mohamed,
  • Suhail Abdullah Alsaqer,
  • Tahir Bashir,
  • Ibrhim. M. A. Suliman

摘要

Overfitting and slow convergence are common problems when using the gradient descent method in neural network training. We use regularization and momentum terms, respectively, to reduce these negative phenomena. In this paper, we consider the convergence of the gradient descent method with smoothing L0 regularization and an adaptive momentum term. The objective function for normal L0 regularization is the sum of a function that is not convex, smooth, or Lipschitz. This makes the error function and the norm of the gradient oscillate. This impediment prevents neural networks from achieving optimal measurement rates for application verification. However, we can address the deficiency of the normal L0 regularization term by using the smoothing approximation techniques. This paper presents the results of weak convergence for smooth L0 regularization using the adaptive momentum method. Furthermore, we have proven the strong convergence results of the theorems. Simulations based on three learning problems—parity problems, function approximation problems, and classification tasks—illustrate the viability of the suggested approach under these circumstances. The suggested approach assumes the momentum coefficient, regularization parameter, and learning rate to be constants. We selected these problems because their unique error surfaces provide an appropriate setting for evaluating the efficacy of the suggested approach. Simulation examples demonstrate the superiority of the suggested algorithm and bolster the theoretical analysis.