Entropy, mutual information, and relative entropy are work horses in Shannon theoretic analyses for communications and compression. These quantities have already been utilized in the analyses of agent learning and the exploration of structure in sequences. This chapter provides concise treatments of these topics to establish notation and to summarize the basic results so that they are available for deeper study in later chapters. There is a beauty in these fundamental results that motivates further study and investigations. As an initial sequence model, we consider a discrete random variable U that takes on the values \(\{1, 2, \dots , M\}\) , where the set of possible values of U is often called the alphabet, denoted as U, and the elements of the set are called letters of the alphabet.