Information Theory of Complex Systems

What do we mean, when we say that a given system shows “complex behavior”, can we provide precise measures for the degree of complexity? This chapter offers an account of several common measures of complexity together with the relation of complexity to predictability and emergence. Following a self-contained introduction to information theory and statistics, we will learn about probability distribution functions, Bayesian inference, the law of large numbers, and the central limit theorem. Next, Shannon entropy and mutual information will be discussed, two concepts that play central roles both in the context of time series analysis, and as starting points for the formulation of quantitative measures of complexity. The chapter concludes with a short overview regarding generative approaches to complexity.