Simple variational inference based on minimizing Kullback–Leibler divergence

We introduce a new methodology of estimation of the true distribution. The procedure of getting estimated distribution is constructed from Bayesian statistical models in which the statistical model is fixed and prior distributions for the parameters are varied. Then we consider the Kullback–Leibler divergence of the true distribution from the estimated one and derive variational formulae of the Kullback–Leibler divergence over prior distributions. Next, we propose a Newton–Raphson method for simulating the prior distribution, which is the critical point, based on the Riemannian geometry of the probability simplex. The method can run once a sample from the true distribution is obtained without any other knowledge of the true distribution. For the geometry, we employ the Riemannian metric induced from the characteristic function, which appears in Vinberg’s theory of homogeneous convex cones. As a by-product of the geometry, we derive an interpretation that the Kullback–Leibler divergence is the logarithm of a gauge transformation. From this, we obtain a viewpoint that envisaging the true distribution is nothing but gauge fixing, and this depiction as gauge theory seems to match the context of statistics. Also, we show some numerical results on how well our methodology works.