Comprehensive Performance Analysis of PySpark and Pandas for Classification and Clustering Task
摘要
This article focuses on the comprehensive evaluation of performance across PySpark and Pandas, employing machine learning algorithms for clustering and classification. We have also implemented a data pipeline in PySpark. For the clustering task, we have applied K-Means and Gaussian Mixture Model (GMM) on the SDSS15 dataset using both PySpark and Pandas. The work compares training time and Silhouette scores, shedding light on the effectiveness and performance of each approach. In the classification domain, Logistic Regression, Decision Tree, Random Forest, and Gradient Boosting Tree (GBT) algorithms are applied to the Credit Card Fraud Detection dataset using PySpark and Pandas libraries. Here, training time and classification accuracies are compared, providing the strengths and weaknesses of each implementation. The finding from this study sheds light on the various trade-offs that exist between the PySpark and Pandas libraries and offers useful details regarding the benefits and drawbacks of each tool for handling challenging machine learning tasks. Considering all, this work offers comprehensive guidance to practitioners searching for optimal algorithms for clustering and classification tasks in a single-node environment.