Physics and Information Theory

This chapter starts with a short introduction of some basic thermodynamic concepts and laws. This is followed by an example involving the mixing of ideal gases, where the change in information is examined alongside thermodynamic properties such as entropy change and work. This serves to highlight the link between entropy in information theory and thermodynamics. In statistical mechanics, a physical system can be characterised by certain macroscopic properties, expressed as expectation values for variables like the internal energy of the system or the number of molecules of a certain type. We demonstrate how the maximum entropy formalism can be applied to such a situation to establish the probability distribution that characterises the system—the Gibbs distribution. It is shown how the formalism can be used to derive a number of thermodynamic relations. As a specific example, for which there is a strong connection to information theory, we explore one-dimensional spin systems. Such a system can be regarded as a symbol sequence generated by a stochastic process, allowing us to directly employ the methodology from Chap.  3 . The entropy based on the internal statistics from such a symbol sequence corresponds to the statistical mechanics entropy of the system (factoring in Boltzmann’s constant). Consequently, we can utilise the maximum entropy formalism on the symbol sequence in order to determine the equilibrium characteristics of the corresponding spin system. The one-dimensional Ising model serves as an illustrative example. Finally, we make an information characterisation of a reversible spin system dynamics in the form of a cellular automaton. It is demonstrated how the approach towards equilibrium can be understood despite the fact that microscopic reversibility conserves the entropy of the system.