The emergence of the Big Data era has made matters more problematic, since the number of datasets has turned popular techniques designed to reduce the impact of class skew impractical. In Big Data context, which necessitates specialized systems with high-performance processing capabilities, this becomes much more challenging. Class imbalance is a prevalent issue in both traditional and Big Data classification that needs to be handled. Machine learning algorithms may be biased in predicting the majority class in Big Data if there is a significant class gap between the majority and minority classes. When it comes to data processing and analysis in Apache Spark, imbalanced data can lead to several issues. It is critical to use sampling approaches to handle uneven data to minimize these problems. In this work, the degree of data imbalance is compared using logistic regression algorithms in Apache Spark framework. The experimental findings demonstrate the importance of data imbalance and how well it can affect the stability and performance of the Apache Spark application.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Impact of Imbalanced Data on Apache Spark

  • Thet Hsu Aung,
  • Aye Myat Myat Paing

摘要

The emergence of the Big Data era has made matters more problematic, since the number of datasets has turned popular techniques designed to reduce the impact of class skew impractical. In Big Data context, which necessitates specialized systems with high-performance processing capabilities, this becomes much more challenging. Class imbalance is a prevalent issue in both traditional and Big Data classification that needs to be handled. Machine learning algorithms may be biased in predicting the majority class in Big Data if there is a significant class gap between the majority and minority classes. When it comes to data processing and analysis in Apache Spark, imbalanced data can lead to several issues. It is critical to use sampling approaches to handle uneven data to minimize these problems. In this work, the degree of data imbalance is compared using logistic regression algorithms in Apache Spark framework. The experimental findings demonstrate the importance of data imbalance and how well it can affect the stability and performance of the Apache Spark application.