Calculus and Optimisation for Machine Learning

This chapter delves into the fundamental concepts of calculus and optimisation related to machine learning, offering both theoretical insights and practical usecases. Starting with the motivation behind using calculus in machine learning, the chapter systematically introduces the concept of limit, which lays the foundation for understanding derivatives and their properties. The discussion on derivatives extends to their role in partial differentiation and gradients, both crucial for optimising machine learning algorithms. A significant portion of the chapter is dedicated to optimisation techniques specifically tailored for neural networks. The chapter begins with an overview of learning definitions and their implications for neural network training. This is followed by an examination of constrained versus unconstrained optimisation and an exploration of the complexities in identifying absolute and local minima of functions. The latter part of the chapter is focused on various optimisation algorithms, briefly discussing line search and trust region methods, then transitioning into specific approaches like steepest descent and gradient descent. The importance of selecting an appropriate learning rate is discussed, along with variations of gradient descent (GD) and strategies for choosing the right mini-batch size. The chapter concludes by exploring the connection between Stochastic Gradient Descent (SGD), fractals, and their implications in machine learning, providing a fascinating example of how complexity appear from very easy optimisation problems.