Ensemble Learning Methods: An Empirical Study

Supervised learning algorithms are effective in most application domains, however they have limitations. A single learning model may miss out on some local regions of the feature space, impacting overall performance. Ensemble learning techniques can be helpful here as they bring together a diverse set of learners, ensuring that even if one misses a region of the feature space other members in the ensemble may be able to learn the pattern. It aims to combine the output of the individual learners in such a way that it can yield best possible final prediction. The purpose of the study is to empirically investigate the performance of different ensemble methods in conjunction with hyper-parameter tuning of the predictive models. We perform an experimental investigation with ensemble learning methods namely Bagging, Boosting, Bagging-Boosting and Stacking using different benchmark datasets. The investigation is based on a data-centric supervised ensemble framework comprising of five engines each with its own functionality. Feature engineering and parameter tuning engines are the highlights of the framework, where relevant-independent features are selected and appropriate set of hyper-parameter values for the participating predictive models are experimentally found out. Extensive detailed experiments are conducted on 25 datasets. For all security datasets we indicated the optimal list of features. For Bagging, Support Vector Machine (SVM) emerged as winner for all the dataset. On the other hand, Gradient Boosting (GB) is the winner in case of Boosting, while both Adaboost and Extreme Gradient Boosting (XGB) performs well in Bagging-Boosting methods. The implications of the empirical study is that the quality of the learning process of any predictive model depends on a number of factors such as choice of hyper-parameters, values of the hyper-parameters, quality of the data and also the pre-processing techniques employed. Also, predictive modeling when combined with hyper-parameter tuning can give significantly better results for most of the datasets.