Investigating the Impact of Unsupervised Dimensionality Reduction on Cluster-Based Machine Learning Techniques in Intrusion Detection

The ubiquitous availability of internet technologies, multiple data sources, data-gathering tools, and extensive storage capabilities have contributed to the heterogeneity of datasets characterized by various features. This heterogeneity often led to the inclusion of extraneous features, causing the dimension of datasets to expand superfluously. The expansion in dimension, especially for intrusion datasets, not only amplifies ambiguity in identifying true intrusions but also magnifies the computational overhead, making dimension reduction an imperative step. Dimension reduction helps to transform the dataset to the reduced feature space by applying some sort of projection technique. This paper studies and implements different kinds of unsupervised dimension reduction techniques, such as Principal Component Analysis (PCA), Non-negative Matrix Factorization (NMF), and Independent Component Analysis (ICA), on CICIDS2017 dataset to explore the impact of reduced dimensional data on cluster formation and analyze the performance of cluster-based machine learning models, i.e., K-Means, Agglomerative, and DBSCAN. The quality of clusters has been determined through Silhouette score, Calinski–Harabasz Index, and Davies–Bouldin Index which have been cross verified through Homogeneity, Completeness, and the Rand Index of these models.