Scholarlink

<p>Sharpness-Aware Minimization (SAM) achieves strong generalization performance in deep learning by seeking flat loss regions. However, vanilla SAM computes the perturbation direction from instantaneous stochastic gradients, which contains a full gradient component that may harm generalization. Moreover, existing momentum-based SAM variants rely on fixed momentum coefficients that require careful tuning. In this paper, we propose DualFR, a novel SAM variant that addresses both limitations. The perturbation direction is designed as the difference between the current gradient and the accumulated momentum, effectively extracting the beneficial stochastic noise component. Adaptive momentum derived from the Fletcher-Reeves conjugate gradient parameter is incorporated into both the perturbation computation and parameter update, eliminating the need for momentum hyperparameter tuning. Our theoretical analysis establishes that DualFR guarantees <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\mathcal {O}(\ln T / \sqrt{T})\)</EquationSource> </InlineEquation> convergence under standard non-convex stochastic optimization assumptions. Extensive experiments on MNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet, together with fine-tuning evaluations on Vision Transformers, demonstrate that DualFR matches or surpasses vanilla SAM and state-of-the-art variants across diverse settings, while exhibiting enhanced robustness to label noise and convergence to flatter loss landscapes.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Sharpness-aware minimization with dual adaptive momentum for training deep neural networks

  • 期刊论文

  • 发表日期：

    2026年7月10日

查看全文
  • Beining Wang

摘要

Sharpness-Aware Minimization (SAM) achieves strong generalization performance in deep learning by seeking flat loss regions. However, vanilla SAM computes the perturbation direction from instantaneous stochastic gradients, which contains a full gradient component that may harm generalization. Moreover, existing momentum-based SAM variants rely on fixed momentum coefficients that require careful tuning. In this paper, we propose DualFR, a novel SAM variant that addresses both limitations. The perturbation direction is designed as the difference between the current gradient and the accumulated momentum, effectively extracting the beneficial stochastic noise component. Adaptive momentum derived from the Fletcher-Reeves conjugate gradient parameter is incorporated into both the perturbation computation and parameter update, eliminating the need for momentum hyperparameter tuning. Our theoretical analysis establishes that DualFR guarantees \(\mathcal {O}(\ln T / \sqrt{T})\) convergence under standard non-convex stochastic optimization assumptions. Extensive experiments on MNIST, CIFAR-10, CIFAR-100, and Tiny-ImageNet, together with fine-tuning evaluations on Vision Transformers, demonstrate that DualFR matches or surpasses vanilla SAM and state-of-the-art variants across diverse settings, while exhibiting enhanced robustness to label noise and convergence to flatter loss landscapes.