Smart Data Simplification: A Comprehensive Feature Selection Framework for High-Dimensional Datasets

The escalating volume of data containing numerous variables poses significant challenges for machine learning tasks, necessitating effective feature selection methods. These methods play a crucial role in alleviating computational burdens, enhancing prediction accuracy, and facilitating better data understanding. This paper introduces an innovative feature selection approach that combines Markov Chain, Quadratic Mutual Information (QMI), and Random Forest techniques to address high-dimensional datasets robustly. Our methodology utilizes a Markov Chain-based algorithm for systematic feature identification and subsequent dimensionality reduction. To address non-linear dependencies and enhance interpretability, we employ K-Means binning. QMI captures intricate non-linear relationships, contributing to a refined quadratic term. Concurrently, Random Forest regression evaluates feature importance, and the SelectFromModel technique retains a crucial subset for prediction. The integration of these techniques provides a comprehensive and robust strategy, as demonstrated through comparisons with different methods Recursive Feature Elimination (RFE) and Recursive Feature Addition (RFA) and models (LSTM, Random Forest, XGBoost). In comparison to existing methods, our approach showcases superior model accuracy, interpretability, and efficiency in handling high-dimensional data. This versatile framework serves as a powerful tool for practitioners seeking to optimize predictive models across various domains, thereby making significant contributions to advanced analytics research.