Self-normalized Importance Sampling for Training Gaussian Mixture Model in Bayesian Convolutional Neural Network

The Gaussian mixture model (GMM) is a powerful statistical model known for its effectiveness in capturing complex posterior distributions in Bayesian inference. However, when applied to Bayesian deep learning, GMMs encounter limitations related to the reparametrization trick, which is crucial to enable backpropagation. To overcome this challenge, our proposed method uses self-normalized importance sampling (SNIS) to estimate the gradient of the objective function for training Bayesian convolutional neural networks (BCNNs). Our approach focuses on adopting the reparametrizable Gaussian components in the GMM as proposal distributions, thus facilitating the functioning of backpropagation. The efficiency and reliability of the resulting estimators are assessed using the effective sample size (ESS) metric, providing insight into their effectiveness in the estimation process. To validate our approach, we implement it on the Bayesian LeNet-5 architecture to classify both the MNIST and Fashion MNIST datasets.