In linear models, usually the solutions are required to be sparse since they identify the most important factors that influence the response of the experiment. For this purpose, regularization techniques are used and the choice of appropriate values for the added parameters becomes of dominant importance. However, most of the statistical models employed in practical applications possess a well conditioned design matrix which may not require regularization for its processing. In this work we will extensively study the structure and properties of several design matrices which have a specific correlation structure. We will examine how the correlation of the data set affects the generalized condition number of the design matrix, and we will conclude useful information about the application of regularization for the solution of the least squares problem.