Contrasting the Landscapes of Feature Selection Under Different Machine Learning Models

Feature selection plays a crucial role in improving the performance of machine learning (ML) models for various prediction tasks and in explaining their recommendations. Feature selection can be defined as an optimization problem whose evaluation function calls on an ML algorithm—a method known as the wrapper approach. While a thorough understanding of the landscape of the feature selection problem might help guide the development of efficient evolutionary algorithms and algorithm selection technologies, only a couple of previous studies have explored this problem’s landscape. In addition, only k-nearest neighbors classification is typically used as an ML model. This paper investigates how the choice of an ML model influences the search difficulty of the feature selection problem. Specifically, we examine the feature selection problem with 14 classification datasets and 6 ML models by means of landscape analysis and local optima networks, and we relate them to the performance of three feature selection algorithms. Our findings have important implications for feature selection problems and algorithms.