Scholarlink

<p>Classical conjugate gradient methods rely solely on first-order information, which limits their ability to capture curvature information in nonconvex stochastic optimization. To address this limitation, this paper proposes a stochastic three-term conjugate gradient algorithm incorporating third-order curvature approximation (TASCG). By integrating third-order tensor information into the search direction, the proposed algorithm enhances its ability to capture the local geometry of nonconvex loss landscapes, while satisfying both the sufficient descent property and boundedness conditions without additional assumptions. Under standard assumptions of gradient Lipschitz continuity and bounded variance, we rigorously establish global convergence of the TASCG algorithm and derive a stochastic first-order oracle complexity bound of <InlineEquation ID="IEq1"><EquationSource Format="TEX">\(\mathcal {O}(\epsilon ^{-2})\)</EquationSource></InlineEquation>, which matches the optimal complexity of classical stochastic gradient methods. To further improve gradient estimation accuracy, we incorporate variance reduction techniques into the TASCG framework, resulting in the variant TASCG-VR. Numerical experiments on nonconvex SVM and empirical risk minimization problems demonstrate that the proposed algorithms significantly outperform standard SGD and SVRG in terms of convergence speed and final solution accuracy, while being more robust to step-size selection. This work provides a novel theoretical perspective and algorithmic framework for integrating higher-order geometric information into stochastic conjugate gradient methods.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Stochastic three-term conjugate gradient: a third-order curvature approximation correction algorithmic framework for machine learning

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月22日

查看全文
  • Jiazhen Liu,
  • Gonglin Yuan,
  • Zhongyu Mo

摘要

Classical conjugate gradient methods rely solely on first-order information, which limits their ability to capture curvature information in nonconvex stochastic optimization. To address this limitation, this paper proposes a stochastic three-term conjugate gradient algorithm incorporating third-order curvature approximation (TASCG). By integrating third-order tensor information into the search direction, the proposed algorithm enhances its ability to capture the local geometry of nonconvex loss landscapes, while satisfying both the sufficient descent property and boundedness conditions without additional assumptions. Under standard assumptions of gradient Lipschitz continuity and bounded variance, we rigorously establish global convergence of the TASCG algorithm and derive a stochastic first-order oracle complexity bound of \(\mathcal {O}(\epsilon ^{-2})\), which matches the optimal complexity of classical stochastic gradient methods. To further improve gradient estimation accuracy, we incorporate variance reduction techniques into the TASCG framework, resulting in the variant TASCG-VR. Numerical experiments on nonconvex SVM and empirical risk minimization problems demonstrate that the proposed algorithms significantly outperform standard SGD and SVRG in terms of convergence speed and final solution accuracy, while being more robust to step-size selection. This work provides a novel theoretical perspective and algorithmic framework for integrating higher-order geometric information into stochastic conjugate gradient methods.