Gradient descent optimization algorithms are the most popular and one of the most used tools when training machine learning models. Different variants of gradient descent each have their strength and weaknesses. Focusing on batch gradient descent optimization, this article aims to provide the reader with an overview of how changing the parameters can change the algorithm’s behavior and cause undesirable outcomes. Numerical computation on specific functions is used to provide a more intuitive view.