Capacity of Neural Networks

Chapter 5 presented several notions of capacity for a linear separation unit, which includes the single neuron, with its dot-product threshold. This chapter shows how to generalize the memory-equivalent capacity of a single neuron to a network of neurons. This question has sparked many approaches in the machine learning community, including fields such as “neural architecture search” or the wider field of “hyper parameter optimization (HPO).” However, as explained in Chap. 5 , two models of the same capacity are able to represent the exact same functions. The understanding of how there can be an analytic solution to neural network capacity is facilitated by remembering the history of the neuron as an (bio-) electrical circuit element, functioning as an energy threshold. Early implementations of neurons (Widrow and Hoff 1960) where hardware-only and neural networks were therefore electrical circuits designed by electrical engineers. In electrical engineering, understanding networks of electrical components analytically, usually by seeing them as connected in series or in parallel, is standard theory and practice. It should therefore be no surprise that neural networks can be understood in the same way. Fortunately, information theory allows us to mathematically abstract from having to go back to electrical engineering.