Scholarlink

<p>Momentum techniques and the Powerball function have been proven effective in stochastic optimization algorithms, widely utilized in large-scale optimization scenarios. Nonetheless, the integration of these methodologies into stochastic optimization algorithms and the determination of their initial learning rates persist as unresolved and pivotal issues. In this study, we integrate momentum techniques and the Powerball function into the SARAH (StochAstic Recursive grAdient algoritHm), culminating in the inception of a novel variance-reduced gradient descent algorithm named PM-SARAH. Moreover, a pair of adaptive step size variants are respectively integrated into the outer and inner loops of PM-SARAH, giving rise to PM-SARAH-AS and PM-SARAH-RAS. Ultimately, through comparative experimentation with state-of-the-art optimization algorithms on standard machine learning tasks and certain non-convex scenarios, the empirical results underscore the superior performance of the algorithms elucidated in this paper.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A stochastic recursive gradient algorithm integrating momentum and the powerball function with adaptive step sizes

  • 期刊论文

  • 发表日期：

    2025年1月19日

查看全文
  • Chuandong Qin,
  • Zilin Cai,
  • Yuhang Guo

摘要

Momentum techniques and the Powerball function have been proven effective in stochastic optimization algorithms, widely utilized in large-scale optimization scenarios. Nonetheless, the integration of these methodologies into stochastic optimization algorithms and the determination of their initial learning rates persist as unresolved and pivotal issues. In this study, we integrate momentum techniques and the Powerball function into the SARAH (StochAstic Recursive grAdient algoritHm), culminating in the inception of a novel variance-reduced gradient descent algorithm named PM-SARAH. Moreover, a pair of adaptive step size variants are respectively integrated into the outer and inner loops of PM-SARAH, giving rise to PM-SARAH-AS and PM-SARAH-RAS. Ultimately, through comparative experimentation with state-of-the-art optimization algorithms on standard machine learning tasks and certain non-convex scenarios, the empirical results underscore the superior performance of the algorithms elucidated in this paper.