Scholarlink

<p>Stochastic gradient descent (SGD) underpins deep learning, yet traditional continuous-time analyses rely on the population gradient assumption, ignoring mini-batch noise and yielding impractical generalization bounds that contradict observed stable gaps in overparameterized models. We propose an empirical gradient-driven continuous-time SGD framework, which captures mini-batch noise via an Itô diffusion with empirical-gradient drift and diffusion. For the generalization gap (population vs. empirical risk), we derive three core results: (1) high-order moment bounds growing as <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(O\left( (1 + t^p + t^{p/2})^{1/p}\right) \)</EquationSource> </InlineEquation>; (2) time-uniform concentration bounds covering 95–100% of gap variations at 95% confidence; (3) ergodic convergence of the time-averaged gap to a stationary limit at an <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(O(1/\sqrt{T})\)</EquationSource> </InlineEquation> rate (experimental <InlineEquation ID="IEq3"> <EquationSource Format="TEX">\(R^2 &gt; 0.98\)</EquationSource> </InlineEquation>). Validated on three stratified datasets (CIFAR-10-5k, SVHN-1k, CIFAR-100-10k) using a ResNet-18 architecture, our theory aligns with empirical observations. Leveraging these results, we design an adaptive training strategy that reduces training epochs by 28–40%, cuts the final generalization gap by 13–19% compared to fixed-epoch training, and preserves test accuracy within 0.5 percentage points of the patience-based baseline. By grounding SGD theory in empirical dynamics, we bridge the gap between theory and practice, providing a reproducible, data-driven approach to efficient training while advancing understanding of SGD’s generalization behavior.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Empirical Gradient-Driven Continuous-Time SGD: Generalization Gap Dynamics and Practical Adaptive Training

  • 期刊论文

  • 发表日期：

    2026年4月3日

查看全文
  • Kai Cui,
  • Wan Li

摘要

Stochastic gradient descent (SGD) underpins deep learning, yet traditional continuous-time analyses rely on the population gradient assumption, ignoring mini-batch noise and yielding impractical generalization bounds that contradict observed stable gaps in overparameterized models. We propose an empirical gradient-driven continuous-time SGD framework, which captures mini-batch noise via an Itô diffusion with empirical-gradient drift and diffusion. For the generalization gap (population vs. empirical risk), we derive three core results: (1) high-order moment bounds growing as \(O\left( (1 + t^p + t^{p/2})^{1/p}\right) \) ; (2) time-uniform concentration bounds covering 95–100% of gap variations at 95% confidence; (3) ergodic convergence of the time-averaged gap to a stationary limit at an \(O(1/\sqrt{T})\) rate (experimental \(R^2 > 0.98\) ). Validated on three stratified datasets (CIFAR-10-5k, SVHN-1k, CIFAR-100-10k) using a ResNet-18 architecture, our theory aligns with empirical observations. Leveraging these results, we design an adaptive training strategy that reduces training epochs by 28–40%, cuts the final generalization gap by 13–19% compared to fixed-epoch training, and preserves test accuracy within 0.5 percentage points of the patience-based baseline. By grounding SGD theory in empirical dynamics, we bridge the gap between theory and practice, providing a reproducible, data-driven approach to efficient training while advancing understanding of SGD’s generalization behavior.