Scholarlink

<p>Second-order methods can accelerate deep neural network training, but their adoption is limited by the cost and instability of estimating and inverting curvature matrices. We revisit Kronecker-factored Fisher approximations via an empirical structural analysis of activation and gradient statistics in modern architectures. Across a range of models, we find that activation statistics capture most of the effective curvature directions, while gradient statistics mainly act as a global or diagonal rescaling. Based on this observation, we propose <i>mean activation curvature</i>, a scalable curvature surrogate that yields two optimizers, <span>MAC</span> and <span>SMAC</span>, offering different trade-offs between expressiveness and efficiency. We further extend the construction to self-attention layers with a structured approximation that retains the role of attention scores, and provide a convergence analysis under standard assumptions. Experiments on vision and language models show that <span>MAC</span>/<span>SMAC</span> matches or improves the accuracy of existing second-order baselines while reducing training time and memory usage. Code: <a href="https://github.com/hseung88/mac">github.com/hseung88/mac</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Mean activation curvature for scalable second-order optimization in deep networks

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月31日

查看全文
  • Hyunseok Seung,
  • Jaewoo Lee,
  • Hyunsuk Ko

摘要

Second-order methods can accelerate deep neural network training, but their adoption is limited by the cost and instability of estimating and inverting curvature matrices. We revisit Kronecker-factored Fisher approximations via an empirical structural analysis of activation and gradient statistics in modern architectures. Across a range of models, we find that activation statistics capture most of the effective curvature directions, while gradient statistics mainly act as a global or diagonal rescaling. Based on this observation, we propose mean activation curvature, a scalable curvature surrogate that yields two optimizers, MAC and SMAC, offering different trade-offs between expressiveness and efficiency. We further extend the construction to self-attention layers with a structured approximation that retains the role of attention scores, and provide a convergence analysis under standard assumptions. Experiments on vision and language models show that MAC/SMAC matches or improves the accuracy of existing second-order baselines while reducing training time and memory usage. Code: github.com/hseung88/mac.