The data used in Machine Learning (ML) algorithms, even if it looks squeaky clean, the ML model may still give us weird predictions. A high number of attributes leads to better models, but it is quite the opposite. One of the reasons might be irrelevant and highly correlated features being high in number. To tackle this problem, here adapting dimensionality reduction helps us extract the highly relevant features and remove features irrelevant to the current prediction. Dimensionality reduction is of two types: feature selection and feature extraction. This paper presents an exhaustive study of feature selection techniques and helps us understand the three widely used feature selection techniques, Forward feature selection, Backward feature extraction, and Random forests. Here also using the K Nearest Neighbors (KNN) classification algorithm in combination with the feature selection techniques. The performances of each classification are studied and compared. Comprehensive experiments of feature selection techniques exposed that they improve classification accuracy.