Consistent Definition of Correlations and Multivariate Mutual Information from Information Theory

It is widely accepted that the correlations of a bivariate distribution are appropriatelyInformation Theory measured by the mutual information, which is the difference between the amounts of information conveyed by either the distribution itself or its marginal distributions. Similarly, when studying the trivariate distributions versus the univariate distributions, the total correlation between the three random variables is measured by the total mutual information. It is also the difference between the information contained in the distribution of the three variables and the information contained in the three univariate marginal distributions. Applying the same ideas to the study of a trivariate distribution versus the bivariate distributions leads to defining the ternary correlation as the difference between the information contained in the distribution of the three variables and the information contained in the three bivariate marginal distributions. This latter information can be evaluated as the SMI for the trivariate distribution, which does not contain more information than the three bivariate marginal distributions; unfortunately, there is no analytical solution to this problem. Two numerical methods are described and compared. Instead of this information-theoretic solution, an analytical simple expression known as the Kirkwood superposition approximationKirkwood superposition approximation (KSA) is also available. It is a compact formula linking the trivariate distribution function with the three bivariate and the three univariate distribution functions. However, this expression does not respect the normalization of the probabilityProbability distribution to unity, and it leads to absurdities in certain cases. Moreover, in this view, the measure of the correlation between N variables is not the sum of correlation terms of different orders, as it is when correlations are defined as the difference between the information contained in the distribution of all the variables and the information contained in its marginal distributions.