Experimental Studies of New Algorithms with Optimism and Exponential Averaging for Finding Saddle Points

Minimax optimization is one of the most important components of machine learning and has lots of applications. However, despite of the sufficient study in this area, some troubles for the stochastic case still remain. One of them is connected to the stochastic saddle point problem: stochastic gradient methods are sensitive to the noise, which can lead even to the divergence of the method. Recently a new approach was proposed: in order to decrease the impact of the noise exponential moving average for the stochastic gradients was used. Based on this approach two new algorithms were invented: Omega and Omega M. The purpose of this article is to experimentally test whether these recently proposed algorithms behave like the well-known stochastic optimistic gradient method for finding saddle points or not and to provide motivation to the authors to continue working on proving the convergence of these algorithms. As a test problem, the training of GANs was chosen, previously represented as the stochastic saddle point problem.