Enhancing the Performance of PSO Algorithm for Clustering High-Dimensional Data Using Autoencoders

The emergence of big data has brought new challenges in processing and analyzing large and complex datasets due to their high dimensionality. Unsupervised learning techniques like clustering have become powerful tools for identifying patterns and relationships in data without the need for labeled examples. One popular unsupervised data clustering technique is K-means and particle swarm optimization (PSO). Combining both techniques’ benefits, K-means clustering with PSO optimization can produce better clustering outcomes. The Elbow approach, which offers the K value for implementing K-means and PSO, automates the data clustering. Clustering high-dimensional data can be challenging due to the curse of dimensionality, where the number of dimensions dramatically outnumbers the number of data points. Therefore, a dimensionality reduction technique must be employed to enhance the performance of clustering high-dimensional data. Thus, we used autoencoder as one of the dimensionality reduction techniques with K-means and PSO clustering and compared the clustering performance on reduced and original data. To implement this method, we used the Nifty 100 (1 min) stock market dataset from the Kaggle website.