The Entropy

In this chapter, we consider random experiments, i.e., experiments for which, on the one hand, we know exactly what results are possible, but on the other hand, we cannot predict an outcome exactly when conducting the experiment, but only with a certain probability. The last section described such an experiment by examining a specific point in a German text to see what character is there. We based this on an alphabet of 30 different characters; thus, for this experiment, there were 30 different results and the corresponding probabilities available. This example shows that the term experiment is very broadly defined and not limited to scientific experiments. The reception of a message is also considered an experiment in this context, with the result of the experiment being the message itself. The special feature of the random experiments in this chapter is that the non-empty set of possible results, always denoted by Ω, may only contain a finite number or countably infinite number of elements. Thus, there is a subset \(N\subseteq\mathbb{N}\) such that a bijection \(N\to\Omega\) exists. If now for each \(\omega\in\Omega\) the probability \(\mathbb{P}(\{\omega\})\) is known that we obtain ω as the result of the random experiment, then we can assign a probability to each subset \(A\subseteq\Omega\) of Ω by \(\mathbb{P}(A):=\sum_{\omega\in A}\mathbb{P}(\{\omega\})\) that the result of the random experiment is in the set.