OAGRE: Outlier Attenuated Gradient Boosted Regression

Gradient boosting is a general machine learning technique for iteratively building a model by fitting the residuals of an existing model. In spite of its genrality as a technique, there are several reasons why gradient boosting may fail to adequately model a given dataset. One of these reasons is the presence of excessive noise in the training data. In this paper we propose a technique that will mitigate the impact of noise under some conditions of heteroscedasticity. We break the assumption of independently distributed noise and derive a simple variation of the gradient boosting algorithm we name Outlier Attenuated Gradient Boosting Regression (OAGRE). We conduct experiments by generating large volumes of synthetic datasets with varying degrees of heteroscedasticity and benchmark our approach against multiple standard gradient boosting algorithms as well as simpler tree models. Our approach provides value in approx 30% of these synthetic datasets with an average improvement of between 7 and 12% lower root mean squared error. Based on these results we recommend the inclusion of the OAGRE method when testing algorithms on data with suspected heteroscedasticity issues.