How to Turn a Leaky Learner into a Sealed One

The outstanding performance of deep neural networks often comes at the expense of a lack of explainability of the results. In this paper, we investigate mutual information between network layers as an information-theoretic means to understand the learning process better. When considering network layers as high-dimensional continuous random variables, the computation of mutual information is a challenging problem. We focus on an approximation method provided by Rényi’s matrix-based entropy functional and evaluate it in a classification task with a multi-layer perceptron. We validate the approximation by checking the data processing inequalities. Furthermore, we use mutual information to detect data leaks, i.e., a loss of information between layers. Sealing such leaks at the beginning of training improves the network’s performance in our classification experiments.