Spawning Gradient Descent (SpGD): A Novel Optimization Framework for Machine Learning and Deep Learning
摘要
First-order optimization methods that leverage gradient information are fundamental for solving problems across diverse domains due to their scalability and computational efficiency. Despite their effectiveness, traditional methods like Gradient Descent (GD) often face challenges related to noise, scalability, and convergence to local optima. This paper introduces Spawning Gradient Descent (SpGD), a novel algorithm that enhances gradient-based optimization by selecting appropriate starting points, dynamically adjusting the learning rate through the proposed Augmented Gradient Descent (AGD) algorithm, and optimizing movement patterns. The AGD mechanism enables dynamic learning rate adjustment by comparing the gradient signs at the newly generated point with those at the current point for each dimension, allowing the optimization process to adapt to the current search state. These innovations mitigate zigzagging, improve initial positioning, and eliminate the need for manual learning rate tuning. By incorporating controlled randomization, SpGD addresses key limitations of traditional methods. Experimental results demonstrate that SpGD achieves enhanced accuracy, resolves step-size reduction constraints, and generates randomized points with superior efficiency. Even with modest computational resources, SpGD facilitates improved exploration of the solution space and increases the precision of locating global minima, despite computational challenges. The Spawning Gradient Descent (SpGD) algorithm demonstrates superior performance across both convex and non-convex benchmarks, significantly outperforming optimizers like GD, Adam, RAdam, and AdaBelief. For instance, on the Quadratic function, SpGD achieves a near-zero performance error of 1.7e