Complexity of Machine Learning

Without doubt, the brain is the most complex adaptive system known to humanity, arguably also a complex system about which we know little. In both respects, the brain faces increasing competition from machine learning architectures. We present an introduction to basic neural network and machine learning concepts, with a special focus on the connection to dynamical systems theory. Starting with point neurons and the XOR problem, the relation between the dynamics of recurrent networks and random matrix theory will be developed. The somewhat counter-intuitive notion of continuous numbers of network layers is shown next to lead to neural differential equations, respectively for information processing and error backpropagation. Approaches aimed at understanding learning processes in deep architectures make often use of the infinite-layer limit. As a result, machine learning can be described by Gaussian processes together with neural tangent kernels. Finally, the distinction between information processing and information routing will be discussed, with the latter being the task of the attention mechanism, the core component of transformer architectures.