Variable Selection

Variable selection is an art. This chapter aims to help you understand the different variable selection techniques that can be used to select the best features in your dataset. It is one of the critical processes in data science. To put it simply, say you are the coach of a soccer team. You want to pick the best team to win the World Cup. You need the best player in each position (best features), and you don’t want too many players who play the same position (multicollinearity) at a given time. In PySpark, there are multiple ways to accomplish this task. This chapter goes through some built-in and custom-built variable selection techniques (see Figure 4-1).