This paper systematically investigates deterministic chaos as a source of structured gradient perturbations for modern optimisers. Seven chaotic maps—including cubic, logistic, single- and multiparametric tent, Hénon, Ikeda and Baker—are projected to bounded, zero-mean signals and injected at principled loci within SGD, Adam and RMSProp. We log each optimiser’s internal state—momentum_buffer, first and second moments, and square_avg—to explain how chaos reshapes exploration and variance control. Experiments on a synthetic regression task show that cubic-map noise accelerates momentum build-up in SGD, Hénon noise yields faster damping of RMSProp’s variance estimate, and tent-map perturbations modulate Adam’s learning rate without destabilising convergence. Parameter sweeps reveal that small chaos scales can even reduce adaptive-variance inflation below the clean baseline. Our analysis highlights the importance of map choice, invariant-measure bias and injection locus, providing concrete design guidelines and opening a path toward chaos-aware optimisation on complex deep-learning benchmarks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Harnessing Deterministic Chaos for Adaptive Gradient Optimization

  • Kenneth Brezinski,
  • Ken Ferens,
  • Witold Kinsner

摘要

This paper systematically investigates deterministic chaos as a source of structured gradient perturbations for modern optimisers. Seven chaotic maps—including cubic, logistic, single- and multiparametric tent, Hénon, Ikeda and Baker—are projected to bounded, zero-mean signals and injected at principled loci within SGD, Adam and RMSProp. We log each optimiser’s internal state—momentum_buffer, first and second moments, and square_avg—to explain how chaos reshapes exploration and variance control. Experiments on a synthetic regression task show that cubic-map noise accelerates momentum build-up in SGD, Hénon noise yields faster damping of RMSProp’s variance estimate, and tent-map perturbations modulate Adam’s learning rate without destabilising convergence. Parameter sweeps reveal that small chaos scales can even reduce adaptive-variance inflation below the clean baseline. Our analysis highlights the importance of map choice, invariant-measure bias and injection locus, providing concrete design guidelines and opening a path toward chaos-aware optimisation on complex deep-learning benchmarks.