Probabilistic Models with Invariance

This proof-of-concept work generalizes the concept of invariance, as used in contrastive learning, to fully probabilistic models (such as, e.g., mixture models) that explicitly describe data distributions in an interpretable fashion, and whose main applications are density estimation (e.g., outlier detection), sampling and tractable inference. Invariance allows allows probabilistic models to operate at a lower effective model complexity, and therefore to deal with more complex (image) data. In this article, we propose iGMM, a Gaussian Mixture Model (GMM) that explicitly incorporates invariance into its loss, which is a generalization of the conventional GMM log-likelihood. When constructing hierarchies of conventional GMM and iGMM instances, we obtain invariance properties that are reminiscent of simple and complex cells in the mammalian visual cortex. We show, by experiments on the MNIST and FashionMNIST dataset, that GMM-iGMM hierarchies can faithfully sample from learned data distributions even if the iGMM is invariant to some aspects of the data, and demonstrate that outlier detection performance is strongly enhanced in GMM-iGMM hierarchies.