Using Mutual Information in Linguistics, Cryptography, and Steganography

In a seminal 1951 articleCryptography, on written EnglishSteganography, Shannon definedF (Anonymous) F2, a measure of the amount of information needed to determine the letter y when x is already known. Comparing F2 with the case where no previous letter is known (F1) leads to considering their difference (F2 − F1) as a (positive) indicator of the influence of one letter on the occurrence of the next letter. This indicator is actually identical to the mutual information MI. If we consider three letters, the measure of the influence of the first two letters on the third letter, which is not due to binary correlations, is F3 − F2. We prove that this difference is positive and we compare a symmetrized version of F3 − F2 with the mutual information MI(3), which was identified in Chap. 5 as the best indicator of ternary correlations among three unordered variables. The relationships between MI(3) and (F3 − F2)Symmetrized are analyzed. The mutual information MI is successively applied to variables referring to letters, letter pairs or triplets, words, and word sequences, e.g., to identify the collocationsCollocation in a language. It is also applied to phonemesPhonemes and sequences of phonemesPhonemes in spoken language. For example, MI is used to study the decrease in correlation between two phonemesPhonemes separated by 0, 1, 2, … other phonemes. We also explain Mutual Information Analysis, a special cryptographic technique used to evaluate the resistance of a cryptographic implementation to a particular type of attack. Finally, some detection techniques for linguistic steganographySteganography are reviewed.