Regression and Variable Selection

Regression is used to explain the relationship between a variable Y , called the response, and a set of one or more variables \(X_1, X_2, \dots , X_N\) , called explanatory variables. The relationship between the response and the predictors is modelled by a function. Regression is the procedure by which the user hypothesize a function describing and explaining the behaviour of the observed data and then estimate the parameters of this function. In this chapter, we present three approaches to data regression: (i) the ordinary least square approach (also known as “frequentist approach”), (ii) Bayesian regression (known as “probabilistic approach”), and (iii) robust regression. Different regression methods are suitable in different contexts to solve different problems, such as the treatment of multicollinear data and the presence of outliers, two typical problems that we will mention in this chapter. Then we will deal with regression diagnostic and variable selection. Regression diagnostics is a subset of regression analysis that investigates if the calculated model and the assumptions we made about the data and the model are consistent with the observed data. Variable selection is the process adopted to (i) make the model easier to interpret, removing redundant non-informative variables; (ii) reduce the size of the problem to enable algorithms to run faster; and (iii) reduce the overfitting and, consequently, to make the model explicative and predictive.