Gradient Overdrive: Avoiding Negative Randomness Effects in Stochastic Gradient Descent

When training a small MLP network, we observed that weights “scrabble” around some time-local mean values before turning in the right direction. This phenomenon also exists for more complex networks. Its negative effect causes an extension of the learning time. This work aims to develop a new method that maximally reduces the phenomenon of scrabbling weights in modern Deep Neural Network (DNN) architectures without losing the positive generalization characteristics of Stochastic Gradient Descent (SGD). The method introduces trimming of gradients when their estimated variance gets too large. The experiments showed tuning the proposed method called Gradient Overdrive (GO) and proved its effectiveness compared to using only a learning optimizer. This comparison clearly shows speeding up the learning process in the early epochs.