Applications of SMI to Communication Systems

The mathematician and informationEntropyand information theorist Claude Shannon decided to use the term “entropy” to denote his newly suggested measure of information developed for telecommunication. For many years, the only relation to the thermodynamic concept sharing the same name was merely a similarity in mathematical notation. However, more recently, it has been understood that Shannon’s measure of information (SMI) has indeed an interpretation also when it comes to thermodynamic systems. This chapter introduces SMI using the intuition that governed Shannon to obtain his powerful results. The principal parts of a communication system are described as well as the statistical modeling of signal sources. It is shown how SMI underlies the theoretical limits for the rate of transmission as well as its quality. At the end of the chapter, we reconnect with thermodynamics through a (toy) example describing energy distributions of particles, applying a result from information theoryInformation Theory to highlight a postulate from Boltzmann.