This article focuses on the comprehensive evaluation of performance across PySpark and Pandas, employing machine learning algorithms for clustering and classification. We have also implemented a data pipeline in PySpark. For the clustering task, we have applied K-Means and Gaussian Mixture Model (GMM) on the SDSS15 dataset using both PySpark and Pandas. The work compares training time and Silhouette scores, shedding light on the effectiveness and performance of each approach. In the classification domain, Logistic Regression, Decision Tree, Random Forest, and Gradient Boosting Tree (GBT) algorithms are applied to the Credit Card Fraud Detection dataset using PySpark and Pandas libraries. Here, training time and classification accuracies are compared, providing the strengths and weaknesses of each implementation. The finding from this study sheds light on the various trade-offs that exist between the PySpark and Pandas libraries and offers useful details regarding the benefits and drawbacks of each tool for handling challenging machine learning tasks. Considering all, this work offers comprehensive guidance to practitioners searching for optimal algorithms for clustering and classification tasks in a single-node environment.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Comprehensive Performance Analysis of PySpark and Pandas for Classification and Clustering Task

  • Rojer Tufani,
  • Snigdha Sen,
  • Pavan Chakraborty

摘要

This article focuses on the comprehensive evaluation of performance across PySpark and Pandas, employing machine learning algorithms for clustering and classification. We have also implemented a data pipeline in PySpark. For the clustering task, we have applied K-Means and Gaussian Mixture Model (GMM) on the SDSS15 dataset using both PySpark and Pandas. The work compares training time and Silhouette scores, shedding light on the effectiveness and performance of each approach. In the classification domain, Logistic Regression, Decision Tree, Random Forest, and Gradient Boosting Tree (GBT) algorithms are applied to the Credit Card Fraud Detection dataset using PySpark and Pandas libraries. Here, training time and classification accuracies are compared, providing the strengths and weaknesses of each implementation. The finding from this study sheds light on the various trade-offs that exist between the PySpark and Pandas libraries and offers useful details regarding the benefits and drawbacks of each tool for handling challenging machine learning tasks. Considering all, this work offers comprehensive guidance to practitioners searching for optimal algorithms for clustering and classification tasks in a single-node environment.