Capacity

In the previous chapter, we have investigated how to estimate the information content of the training data in bits. Since the purpose of a model is to generalize the experimental results into a rule that can predict future experiments, most people would intuitively agree that a model should not be more complex than the training data. We introduced this intuition in Chap. 2 , especially Sect. 2.2.1 . More formally, the information content of the training data should give an upper limit for the complexity of the model. But how does one measure the complexity of a model? To measure the complexity of the model, we need the notion of model capacity, which is the topic of this chapter.