Scholarlink

<p>We introduce Thermodynamic Natural Gradient Descent (NGD-T), an optimizer that enforces a physical speed-cost constraint by combining Fisher-preconditioned updates with a dissipation-aware step-size regulator. While natural gradient methods are known to follow the steepest descent direction in information geometry, we provide a thermodynamic reinterpretation: Natural Gradient Flow uniquely minimizes instantaneous irreversible dissipation for a fixed loss decrease. NGD-T implements this principle in discrete updates by (i) preconditioning gradients with an approximate inverse Fisher, (ii) computing the geometric norm <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\:{{\Delta\:}}_{F}=\nabla\:{L}^{{\top\:}}{F}^{-1}\nabla\:L\)</EquationSource> </InlineEquation>, and (iii) mapping a user-specified dissipation budget <InlineEquation ID="IEq2"> <EquationSource Format="TEX">\(\:{Q}_{\text{b}\text{u}\text{d}\text{g}\text{e}\text{t}}\)</EquationSource> </InlineEquation> to a step size <InlineEquation ID="IEq3"> <EquationSource Format="TEX">\(\:{\eta\:}_{T}\)</EquationSource> </InlineEquation> that saturates the speed-cost bound. We present numerically stable constructions for rank-deficient Fisher estimates, a hybrid nullspace fallback, and scalable K-FAC integration with eigendecomposition caching. On CIFAR-10, ImageNet, and transformer architectures, NGD-T matches or exceeds Adam in convergence while substantially reducing predicted irreversible dissipation and maintaining comparable wall-clock time. NGD-T provides a principled, tunable trade-off between learning speed and thermodynamic cost with theoretical convergence guarantees.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Thermodynamic natural gradient descent (NGD-T) regulates natural-gradient steps by a geometric speed-cost bound

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月1日

查看全文
  • Barco Jie You

摘要

We introduce Thermodynamic Natural Gradient Descent (NGD-T), an optimizer that enforces a physical speed-cost constraint by combining Fisher-preconditioned updates with a dissipation-aware step-size regulator. While natural gradient methods are known to follow the steepest descent direction in information geometry, we provide a thermodynamic reinterpretation: Natural Gradient Flow uniquely minimizes instantaneous irreversible dissipation for a fixed loss decrease. NGD-T implements this principle in discrete updates by (i) preconditioning gradients with an approximate inverse Fisher, (ii) computing the geometric norm \(\:{{\Delta\:}}_{F}=\nabla\:{L}^{{\top\:}}{F}^{-1}\nabla\:L\) , and (iii) mapping a user-specified dissipation budget \(\:{Q}_{\text{b}\text{u}\text{d}\text{g}\text{e}\text{t}}\) to a step size \(\:{\eta\:}_{T}\) that saturates the speed-cost bound. We present numerically stable constructions for rank-deficient Fisher estimates, a hybrid nullspace fallback, and scalable K-FAC integration with eigendecomposition caching. On CIFAR-10, ImageNet, and transformer architectures, NGD-T matches or exceeds Adam in convergence while substantially reducing predicted irreversible dissipation and maintaining comparable wall-clock time. NGD-T provides a principled, tunable trade-off between learning speed and thermodynamic cost with theoretical convergence guarantees.