Enhancing early attack detection: novel hybrid density-based isolation forest for improved anomaly detection

Recently, the frequency and complexity of cyber threats have significantly increased, making it imperative to detect such anomalies in their early stages to minimize harm or data loss. Traditional anomaly detection approaches often prove inefficient in addressing modern, sophisticated threats. To mitigate information risks and negative outcomes, it is essential to detect attacks at an early phase. Many existing AD methods struggle to capture the intricate associations in data visualizations or effectively exploit contextual information for improved performance. In this paper, we propose a Hybrid Density-Based Isolation Forest with Coati Optimization (HDBIF-CO) algorithm for effective anomaly detection (AD) classification, using the NSL-KDD, CICIDS2017, and UNSW-NB15 datasets. The primary objective is to develop a more efficient and accurate method for detecting anomalies and potential cyberattacks in cybersecurity systems. The anomalies are detected through six key stages: the data collection phase, the data preprocessing phase (which involves data normalization and outlier elimination), feature selection, cluster discovery using Density-Based Spatial Clustering of Applications with Noise (DBSCAN), detection using the HDBIF-CO algorithm, and finally, the decision phase. The datasets used—NSL-KDD, CICIDS2017, and UNSW-NB15—contain both anomaly and normal data. During the preprocessing phase, duplicate data are eliminated, and features are extracted using a feature reduction technique to minimize data dimensionality. In the cluster formation phase, clusters are identified, and the HDBIF-CO algorithm is applied to segregate anomalies. The evaluation results demonstrated the reliability and effectiveness of the HDBIF-CO method, achieving 98.9% accuracy, 97.9% precision, 98.5% recall, and a 98.6% F1-score.