The impact of sparsity and entropy criteria on neural network performance

We explore the impact of adding entropy and sparsity criteria to a standard neural network cost function, by considering a variety of network types and applications. Measuring network performance via the testing error, we seek to answer the question: does including an entropy criterion and/or a sparsity criterion with some choice(s) of coefficient(s) produce a performance improvement of the network? The exploration suggests that the addition of a single one of these two criteria, with appropriate choice of coefficient, generates a performance improvement, and the inclusion of both criteria, with appropriate choice of coefficients, generates a further improvement. This suggestion reflects established results for parameter estimation inverse problems in a number of other settings.