Diversified deep hierarchical kernel ensemble regression

Deep ensemble learning models that combine multiple independent deep learning models with multi-layer processing architectures have proven to be effective techniques for improving the accuracy and robustness of deep learning models. In this paper, we propose a diversified kernel ensemble regression method, which is developed from the well-known kernel ridge regression methods. Motivated by multi-view data modeling ideas, we treat each individual kernel as one view of original data in kernel representation space. Therefore, we develop a deep kernel ensemble ridge regression method in Neural Tangent Kernel (NTK) to address the problem faced by traditional kernel ridge regression methods in finding appropriate types of kernels and their parameters. Specifically, as multiple deep kernel regressors share common information from the multi-view kernel representations, our proposed method is built through a hierarchical modeling method, where deep kernel regressors share a common parameter and also have model-specific parameters in individual regressors, which is further helpful in improving our model’s performance. Furthermore, to achieve better diversified deep kernel representations in our proposed method, the Hilbert-Schmidt Independence Criterion (HSIC) is used to regularize our proposed model. In this way, we can find more diversified kernel representations among multiple kernel ensemble regressors to achieve better generalization performance. Experiments on several classification and regression datasets, such as MNIST, TinyImageNet-200, ORL, Credit and YearPredictionMSD demonstrate that our proposed method can achieve best regression performance than other state-of-the-art methods.