<p>Feature selection is a vital step in machine learning tasks, contributing to interpretability, explainability, and dimensionality reduction. Matrix-based techniques are a ubiquitous approach for selecting the most informative features. However, they fail to capture the nonlinear structure of the data. Auto-encoder based methods, on the other hand, are employed for feature selection as they can extract nonlinear patterns from the data. Despite this advantage, existing feature selection auto-encoders are shallow and cannot extract hierarchical deep features. Deep features, however, have the potential to improve the performance of auto-encoders in feature selection tasks. To address this limitation, we propose a Knowledge Distillation Auto-encoder based Feature Selection method that utilizes offline feature-based knowledge distillation as regularization for auto-encoder feature selection. More precisely, incorporating hierarchical information obtained from a deep network can enhance the capability of shallow feature selection algorithms to determine the most complicated and informative features. We evaluated the effectiveness of knowledge distillation on two auto-encoder models for feature selection: (A) Sparse Auto-encoder and (B) Robust Geometric Structure Auto-encoder. To assess the effectiveness of the proposed method, we conducted experiments on ten benchmark datasets and compared our method with state-of-the-art feature selection techniques. Our results demonstrate that this new method achieves significant improvements over some recent techniques.</p>