Feature selection would be a stepping stone in the improvement of model performance, which has been extremely crucial as data grows, both in size and complexity. Present study gives a comparative overview of mathematical techniques concerning mutual information (MI) and conditional mutual information (CMI) methods in feature selection. Although MI measures the degree of correlation between various features, it frequently overlooks potential interactions between them and fails to increase the accuracy. By using non-redundant and informative features, we aim to capture both direct and conditional dependencies while combining MI and CMI. Therefore, to demonstrate its impact on hyperparameter optimization, the study compares MI-only, CMI-only, and coupling MI with CMI approaches on a benchmark dataset. Two distinct perspectives have been used for this comparison: model performance and the efficacy of feature selection. Results show that model’s robustness can be increased by combining MI and CMI, since coupling can identify whole new informative traits for more accurate predictions.