Speeding up the Oscillation-Free Modified Heavy Ball Algorithm

This paper presents modified, faster momentum minimization algorithms based on existing ones. The modified algorithms monotonically decrease the objective function and do not allow it to oscillate. The modification scheme aims to enhance momentum minimizers by incorporating contemporary line search procedures and restarts, akin to the state-of-the-art unconstrained minimizers. We also investigate the unique resource of oscillation-free momentum minimizers for their further acceleration. In particular, the wider range of variation in the friction-related coefficient within the model significantly impacts the performance time. Our previously developed techniques can be used to prove the convergence of modified algorithms. In this paper, we focus on the technical and experimental aspects of these algorithms. To determine the efficiency of the new algorithms, numerical experiments were conducted on standard optimization test functions and on single-layer neural networks for several datasets. Comparisons were made with the best unconstrained minimization algorithms – lcg, L-BFGS and ADAM. Oscillation-free momentum algorithms are significantly easier to design and implement than lcg and L-BFGS, while still being competitive in terms of performance. Collections of minimizers and test functions have been uploaded to GitHub.