Unsupervised and Supervised Deep Learning forAnomaly-Based Intrusion Detection: A Comparative Study of VAE, GAN, and XGBoost
摘要
The increasing complexity and scale of cyber-attacks have made network security a matter of concern in modern computing systems. Traditional signature-based Intrusion Detection Systems (IDS) are effective at detecting known attacks but are redundant when it comes to new or zero-day attacks because they rely on pre-set rules and fixed patterns. The weakness compromises modern infrastructures, making them susceptible to more sophisticated intrusions that evade traditional detection methods. Considering these limitations, this work investigates the application of deep learning techniques to anomaly-based intrusion detection, with an emphasis on models able to identify anomalies from typical traffic behaviour. Specifically, the study compares and evaluates the performance of three models: a Variational Autoencoder (VAE), a Generative Adversarial Network (GAN), and an XGBoost classifier, on the widely employed KDD Cup 1999 benchmark. The comparison between supervised (XGBoost) and unsupervised (VAE and GAN) techniques is particularly pertinent, as it reflects the practical real-world compromise between the availability of labeled data and the need for generalizability in real-world Intrusion Detection System (IDS) installations. The VAE, trained on the full dataset (~4.9 million records), learns the normal traffic latent distribution and identifies anomalies using reconstruction error thresholds. The GAN, trained on a 10% sample, produces data and uses its discriminator to identify anomalies, along with a logistic regression classifier. The XGBoost model, trained on a stratified sample, is a supervised detection baseline, with feature importance analysis providing key intrusion indicators. Empirical outcomes reveal that the VAE recorded 99.69% recall and 98.76% accuracy, while the GAN, under its direct assessment mode, recorded improved performance at 99.27% accuracy and 99.55% F1-score. Under a supervised environment, the XGBoost also showed better classification ability. The unsupervised architecture based deep learning techniques used in this study showed improved resilience and flexibility of anomaly-based IDS systems against cyber-threats.