Information Theory for Lattice Systems

The fundamental concepts of information theory, entropy and relative information, can be used to analyse symbol sequences. In such an analysis, the aim is to decompose the information per symbol into two components: the ordered or redundant part of the information content and the disordered part, the entropy. The former quantifies the information present in correlations between symbols, while the latter quantifies the randomness of the sequence. The formalism is illustrated by examples of symbol sequences generated by Markov processes and Hidden Markov models. To characterise the complexity of a symbol sequence, we can use the concept of “effective measure complexity” or “excess entropy”. This quantifies one aspect of complexity in the sequence and is explored and discussed in detail. Finally, the chapter concludes by presenting a generalisation of the formalism to two-dimensional patterns of symbols.