Theory on Learning

Understanding the learning algorithm of a neural network requires a fair amount of mathematical notation. This chapter describes some relevant theoretical aspects of how neural networks work. We will review the concepts of loss functions and gradient descent. Note that this chapter is by no means a complete description of how networks learn. As indicated at the end of this chapter, many other people have done an excellent job of accurately describing the theoretical underpinnings of learning and optimization mechanisms. Instead, this chapter is intended to reinforce some aspects of the implementation explained in the previous chapters, with the assumption that you are familiar with basic differential calculus.