<p>This paper introduces Javidi, a novel deep learning optimizer that systematically integrates third-moment estimation with Nesterov accelerated momentum and AMSGrad stabilization. While existing adaptive optimizers primarily rely on first- and second-order gradient moments, they largely overlook informative higher-order statistics. Javidi addresses this limitation through three core innovations: (1) dynamic third-moment estimation to capture gradient distribution skewness, (2) a time-decaying weighting coefficient (<InlineEquation ID="IEq1"> <EquationSource Format="TEX">$\lambda _t = \lambda _0 / \sqrt{t}$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <msub> <mi>λ</mi> <mi>t</mi> </msub> <mo>=</mo> <msub> <mi>λ</mi> <mn>0</mn> </msub> <mo stretchy="false">/</mo> <msqrt> <mi>t</mi> </msqrt> </mrow> </math></EquationSource> </InlineEquation>) that adaptively modulates the contribution of the third moment throughout training, and (3) Nesterov-style look-ahead momentum combined with AMSGrad’s maximum second-moment tracking to enhance convergence stability. Extensive empirical evaluation across five benchmark datasets spanning computer vision (MNIST, noisy MNIST, and noisy CIFAR-10) and natural language processing (SST-2 and AG News) demonstrates Javidi’s consistently strong performance compared to five state-of-the-art optimizers, namely Adam, AdamW, Lion, LAMB, and Sophia. All experiments were conducted over 7 independent runs with different random seeds. Results are reported as mean values with 95<InlineEquation ID="IEq2"> <EquationSource Format="TEX">$\%$</EquationSource> <EquationSource Format="MATHML"><math> <mo>%</mo> </math></EquationSource> </InlineEquation> bootstrap confidence intervals (percentile method, 5000 resamples). Statistical significance was assessed using pair-wise t-tests on accuracy values with Bonferroni correction for multiple comparisons. On vision benchmarks, Javidi achieves top-tier accuracy on MNIST (0.9919 [0.9916, 0.9922]) and noisy MNIST (0.9923 [0.9918, 0.9927]), performing statistically equivalently to Lion (<InlineEquation ID="IEq3"> <EquationSource Format="TEX">$p = 0.4649$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi>p</mi> <mo>=</mo> <mn>0.4649</mn> </mrow> </math></EquationSource> </InlineEquation> and <InlineEquation ID="IEq4"> <EquationSource Format="TEX">$p = 0.8344$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi>p</mi> <mo>=</mo> <mn>0.8344</mn> </mrow> </math></EquationSource> </InlineEquation>, respectively) while significantly outperforming other baselines (<InlineEquation ID="IEq5"> <EquationSource Format="TEX">$p < 0.01$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi>p</mi> <mo><</mo> <mn>0.01</mn> </mrow> </math></EquationSource> </InlineEquation>). On the more challenging noisy CIFAR-10 dataset, Javidi ranks second (0.7882 [0.7828, 0.7928]) and significantly outperforms LAMB and Sophia (<InlineEquation ID="IEq6"> <EquationSource Format="TEX">$p < 0.001$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi>p</mi> <mo><</mo> <mn>0.001</mn> </mrow> </math></EquationSource> </InlineEquation>). For NLP tasks, Javidi delivers competitive results on SST-2 (0.9102 [0.9075, 0.9128]) and AG News (0.9101 [0.9092, 0.9107]), maintaining statistical equivalence to leading baselines on SST-2 while significantly outperforming most competitors on AG News (<InlineEquation ID="IEq7"> <EquationSource Format="TEX">$p < 0.001$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mi>p</mi> <mo><</mo> <mn>0.001</mn> </mrow> </math></EquationSource> </InlineEquation>). Across all datasets, Javidi exhibits exceptional stability, with coefficients of variation not exceeding <InlineEquation ID="IEq8"> <EquationSource Format="TEX">$0.97\%$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>0.97</mn> <mo>%</mo> </mrow> </math></EquationSource> </InlineEquation>, indicating low sensitivity to random initialization. The optimizer maintains competitive computational efficiency, incurring a modest overhead typically within 7–<InlineEquation ID="IEq9"> <EquationSource Format="TEX">$15\%$</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mn>15</mn> <mo>%</mo> </mrow> </math></EquationSource> </InlineEquation> relative to Adam, depending on the dataset, while achieving smoother convergence, lower final training losses, and notable robustness to hyperparameter variations. These results show that integrating higher-order gradient moments can significantly improve optimization performance with limited computational overhead. Code and supplementary materials are available at <a href="https://github.com/Aliyar4061/Javidiv2-Optimizer">https://github.com/Aliyar4061/Javidiv2-Optimizer</a>.</p>

Javidi: enhanced deep learning optimization through third-moment integration and adaptive acceleration

期刊论文
发表日期：

2026年6月26日

查看全文

Ali Zeydi Abdian,
Mohammad Masoud Javidi,
Najme Mansouri

摘要

This paper introduces Javidi, a novel deep learning optimizer that systematically integrates third-moment estimation with Nesterov accelerated momentum and AMSGrad stabilization. While existing adaptive optimizers primarily rely on first- and second-order gradient moments, they largely overlook informative higher-order statistics. Javidi addresses this limitation through three core innovations: (1) dynamic third-moment estimation to capture gradient distribution skewness, (2) a time-decaying weighting coefficient ( $\lambda _t = \lambda _0 / \sqrt{t}$ $λ_{t} = λ_{0} / \sqrt{t}$ ) that adaptively modulates the contribution of the third moment throughout training, and (3) Nesterov-style look-ahead momentum combined with AMSGrad’s maximum second-moment tracking to enhance convergence stability. Extensive empirical evaluation across five benchmark datasets spanning computer vision (MNIST, noisy MNIST, and noisy CIFAR-10) and natural language processing (SST-2 and AG News) demonstrates Javidi’s consistently strong performance compared to five state-of-the-art optimizers, namely Adam, AdamW, Lion, LAMB, and Sophia. All experiments were conducted over 7 independent runs with different random seeds. Results are reported as mean values with 95 $\%$ $%$ bootstrap confidence intervals (percentile method, 5000 resamples). Statistical significance was assessed using pair-wise t-tests on accuracy values with Bonferroni correction for multiple comparisons. On vision benchmarks, Javidi achieves top-tier accuracy on MNIST (0.9919 [0.9916, 0.9922]) and noisy MNIST (0.9923 [0.9918, 0.9927]), performing statistically equivalently to Lion ( $p = 0.4649$ $p = 0.4649$ and $p = 0.8344$ $p = 0.8344$ , respectively) while significantly outperforming other baselines ( $p < 0.01$ $p < 0.01$ ). On the more challenging noisy CIFAR-10 dataset, Javidi ranks second (0.7882 [0.7828, 0.7928]) and significantly outperforms LAMB and Sophia ( $p < 0.001$ $p < 0.001$ ). For NLP tasks, Javidi delivers competitive results on SST-2 (0.9102 [0.9075, 0.9128]) and AG News (0.9101 [0.9092, 0.9107]), maintaining statistical equivalence to leading baselines on SST-2 while significantly outperforming most competitors on AG News ( $p < 0.001$ $p < 0.001$ ). Across all datasets, Javidi exhibits exceptional stability, with coefficients of variation not exceeding $0.97\%$ $0.97 %$ , indicating low sensitivity to random initialization. The optimizer maintains competitive computational efficiency, incurring a modest overhead typically within 7– $15\%$ $15 %$ relative to Adam, depending on the dataset, while achieving smoother convergence, lower final training losses, and notable robustness to hyperparameter variations. These results show that integrating higher-order gradient moments can significantly improve optimization performance with limited computational overhead. Code and supplementary materials are available at https://github.com/Aliyar4061/Javidiv2-Optimizer.