Fast weighted linear model trees
摘要
Decision trees are powerful predictive models widely used in machine learning. However, simple trees like CART (Classification and Regression Trees) struggle to capture linear relationships. Linear model trees address this by fitting linear models in the nodes, combining tree structure with linear regression for better interpretability. Recently, a new approach called PILOT (Piecewise Linear Organic Tree) was introduced, offering the speed of traditional decision trees with the flexibility of linear model trees. This paper proposes a weighted extension of the PILOT tree algorithm, allowing its full potential to be exploited in a broader range of applications. During training, each observation can be assigned a weight that reflects its importance in the loss function. Crucially, incorporating these weights does not change the computational complexity of the PILOT algorithm. Three applications demonstrate the benefits of weighted PILOT trees. First, they are used in a one-step boosting model that fits two complementary PILOT trees. This approach strikes a balance between the simplicity of a single PILOT tree and the performance of an entire (random) forest of PILOT trees. Second, weighted PILOT is applied to imbalanced regression tasks. By increasing the weights of underrepresented points, prediction accuracy improves in critical regions. Finally, in the third application, we apply weighted PILOT to the covariance shift problem. Applying weights effectively mitigates covariate shift, leading to improved performance compared to the unweighted version.