In the current era, there has been a significant increase in the utilization of digital devices, highlighting the critical importance of securing the data stored within them. Research initiatives are being conducted to enhance the security of these devices. Nevertheless, with the continuous advancement of machine learning algorithms, there remains a demand for more effective and efficient models. The feature selection technique we have proposed, known as the enhanced smell detection algorithm with modifications (ESDAMs), in conjunction with the CatBoost classifier, significantly enhances the detection of cyber threats relating to intrusion detection and suspicious communications. The datasets utilized include NSL KDD, CICIDS-2017, UNR-IDD, and the CIC-DarkNet 2020 dataset. The ESDAM method selects the most relevant features from the large dataset, leading to reduced computational complexity, and the CatBoost classifier’s iterative approach ensures consistent and higher performance.