Neural Networks

In this chapter, we deal with artificial neural networks. After some simple examples on the exact representation of Boolean functions by neural networks with Heaviside activation, we discuss the uniform approximation of continuous functions by shallow or deep neural networks. Highlights are the theorems of Cybenko, Leshno-Lin-Pinkus-Schocken, and Hanin. In the second part of the chapter, we outline the method of backpropagation, with which the weights and biases of a neural network can be adapted to a given dataset. We first treat deep neural networks with linear output, which can be used, for example, as regressors for continuously labeled data. Subsequently, we consider deep neural networks with softmax output. The latter are well suited for one-hot-encoded datasets, which occur for example in handwriting recognition tasks.