Scholarlink

<p>Stepsize is a key factor for the convergence and efficiency of stochastic gradient descent (SGD) methods. The stochastic Polyak stepsize (SPS) has shown great potential in machine learning problems. In this paper, we propose a new variant of SPS that has the flexible of using cumulative function difference from any previous iteration. It is computed by a lower bound of the infimum of a random function constructed by incorporating the original objective function with the variance reduction technique and momentum. Hence the stepsize is suitable for non-interpolated problems. Based on the stepsize and SGD, we develop an algorithm, namely SPSVRM, that converges to an exact solution at a sublinear rate for convex and non-interpolated problems. Numerical experiments shows that SPSVRM performs better than other SGDs with recent variants of SPS.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Stochastic Polyak stepsize for SGD with variance reduction and momentum

  • 期刊论文

  • 发表日期：

    2025年1月7日

查看全文
  • Chenchen Liu,
  • Yakui Huang

摘要

Stepsize is a key factor for the convergence and efficiency of stochastic gradient descent (SGD) methods. The stochastic Polyak stepsize (SPS) has shown great potential in machine learning problems. In this paper, we propose a new variant of SPS that has the flexible of using cumulative function difference from any previous iteration. It is computed by a lower bound of the infimum of a random function constructed by incorporating the original objective function with the variance reduction technique and momentum. Hence the stepsize is suitable for non-interpolated problems. Based on the stepsize and SGD, we develop an algorithm, namely SPSVRM, that converges to an exact solution at a sublinear rate for convex and non-interpolated problems. Numerical experiments shows that SPSVRM performs better than other SGDs with recent variants of SPS.