Information Theory
摘要
Information theory was introduced by Claude Shannon in his 1948 paper “A Mathematical Theory of Communication” which laid the foundation for all modern digital communications. This chapter introduces the basic concepts of information theory through a series of theorems and proofs. After an introduction to information theory, the remainder of this chapter begins with a treatment of entropy and and mutual information. Discrete random vectors are considered next that are typically found in digital communication systems. Fundamental to information theory as applied to communication is the notion of channel capacity that establishes a fundamental limit on the data rate for reliable communication over a given coding channel. Finally, the above concepts that are developed in the context of discrete random variables are extended to continuous random variables.