Feature Importance and Selection

This chapter offers an in-depth exploration of various methods used to assess feature importance in machine learning models. Initially, it highlights the importance of identifying key features in a machine learning model, using examples from the healthcare and finance sectors to illustrate its significance. The chapter categorises feature importance assessment methods into three types: filter, wrapper, and embedded methods. Filter methods are preprocessing steps that select features independently of the model, with examples that include the variance threshold, the Chi-square test, the information gain, and the correlation coefficient. These methods are notable for their computational efficiency and scalability. The section on wrapper methods delves into techniques like recursive feature elimination, forward feature selection, backward feature elimination, exhaustive feature selection, and information content elimination. These methods are flexible and consider interactions between features, which can lead to better model performance but at the cost of higher computational intensity. Embedded methods, such as LASSO, ridge regression, elastic net, and decision trees, are integrated into the model training process and specific to each model, making them somewhat challenging to interpret in scientific contexts. The chapter also provides a detailed explanation of forward feature selection, backward feature elimination, information content elimination, and permutation feature importance.