Sparse Besov Space Analysis of Representations in Machine Learning

We present a survey of a unified approach that analyzes representation spaces in signal processing, classic machine learning and deep learning using familiar concepts of sparsity and smoothness spaces. The theory is especially suited for high dimensional spaces. It is known that it is not possible to characterize the approximation spaces of deep learning models using classic smoothness spaces [19]. Furthermore, many problems solved by deep learning are high dimensional where classical function spaces such as the isotropic Besov spaces are somewhat inadequate. Here, we shed some light on this problem by analyzing the dynamics of sparse Besov function smoothness of representations across the layers of a deep neural network, during and after training. We justify our approach by extensive experiments demonstrating that in well-performing trained networks, the sparse Besov smoothness of the training set, measured in the corresponding hidden layer feature map representation, increases from layer to layer.