Hyperparameter Optimization for Gradient-Boosted Tree-Based Machine Learning Models and Their Effect on Model Performance

Gradient-boosted tree-based machine learning models have several parameters called hyperparameters that control their fit and performance. Several methods exist to optimize hyperparameters for a given regression or classification problem. However, how and to what extent the tuning of hyperparameters can affect model performance is not well understood. Therefore, we investigated the effect of optimizing the hyperparameters for three commonly used tree-based machine learning models on their performance compared to the default. We evaluated extreme-gradient boosting (XGBoost), light-gradient boosting machine (Lightgbm) and Catboost models for predicting the price of food items listed in an online food delivery application. Mean absolute error, root mean squared error and R-squared served as the evaluation metrics. We found that optimizing hyperparameters consistently improved the performance of the models, regardless of the model type. Also, the model rank depended on whether hyperparameters were optimized or not. These findings suggest that the default settings in common tree-based models should not be relied on for real-world applications. An attempt should be made to optimize hyperparameters for a given model to improve model accuracy. In this chapter, we have evaluated the role of hyperparameter optimization on some of the latest and most advanced tree-based machine learning models. The novelty of our analysis lies in the fact that we demonstrate the role of hyperparameter optimization for advanced use cases such as those involving Catboost-like algorithms.