SMBOX: A Scalable and Efficient Method for Sequential Model-Based Parameter Optimization

The application of Machine Learning (ML) algorithms continues to grow and shows no signs of slowing down. Each ML algorithm has an associated set of hyperparameter values that need to be set to achieve the best performance for each problem. The task of selecting the best parameter values for the problem at hand is known as Hyperparameter Optimisation (HPO). Traditionally this has been carried out manually or by unguided programmatic approaches such as grid or random search. These approaches can be extremely time-consuming and inefficient, especially when dealing with more than a handful of parameters. More advanced methods involving Evolutionary Heuristics [23] or Bayesian Optimisation [17, 28] use a guided search approach and are widely considered as the gold standard approach for hyperparameter optimisation. In this paper, we introduce SMBOX ( https://github.com/smbox/smbox ), a novel HPO search strategy developed to rival the state-of-the-art, SMAC [15]. Our benchmarking on public classification datasets, against both SMAC and a Random search baseline, shows that SMBOX not only challenges SMAC in tuning hyperparameters for two prevalent ML algorithms, but it also excels in finding good hyperparameter values quicker than SMAC. This rapid optimisation capability is extremely powerful, particularly in situations where time or computational resources are constrained or costly.