This chapter surveys a unified framework for the analysis of SGD-type methods for smooth and strongly convex optimization, with the main results based on the unified theory introduced in [21]. Under a simple parametric assumption on stochastic gradient estimators, a wide range of algorithms—including classical SGD, methods with expected smoothness, interpolation regimes, variance reduction, coordinate descent, and distributed methods with compression—can be analyzed within a single convergence proof. The framework yields linear convergence guarantees in expectation and recovers known optimal rates in many standard settings. Extensions, limitations, and alternative analytical approaches are briefly discussed, highlighting the framework’s pedagogical value and its role in systematizing modern stochastic optimization methods.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Unified Analysis of SGD-Type Methods

  • Eduard Gorbunov

摘要

This chapter surveys a unified framework for the analysis of SGD-type methods for smooth and strongly convex optimization, with the main results based on the unified theory introduced in [21]. Under a simple parametric assumption on stochastic gradient estimators, a wide range of algorithms—including classical SGD, methods with expected smoothness, interpolation regimes, variance reduction, coordinate descent, and distributed methods with compression—can be analyzed within a single convergence proof. The framework yields linear convergence guarantees in expectation and recovers known optimal rates in many standard settings. Extensions, limitations, and alternative analytical approaches are briefly discussed, highlighting the framework’s pedagogical value and its role in systematizing modern stochastic optimization methods.