On the effectiveness of partially deterministic Bayesian neural networks

Bayesian neural networks (BNNs) with computationally expensive Hamiltonian Monte Carlo sampling methods are often considered to provide better predictive performance than the maximum a posterior (MAP) solution. Here, as an alternative to sampling all parameters of a BNN (full-random), we experimentally evaluate partially deterministic BNNs that fix some part of the neural network parameters to their MAP solution. In particular, we consider various strategies for fixing half, or all parameters of a layer to the MAP-solution. Over a wide variety of regression and classification tasks, we find that partially deterministic BNNs often significantly improve predictive performance over the MAP-solution, with up to around 24% reduction in negative log-likelihood. Notably, we also find that partially deterministic BNNs that fix half of the parameters in each layer can also reduce under-fitting of full-random BNNs, resulting in up to 7% reduction in negative log-likelihood.