Impact of Imbalanced Data on Apache Spark
摘要
The emergence of the Big Data era has made matters more problematic, since the number of datasets has turned popular techniques designed to reduce the impact of class skew impractical. In Big Data context, which necessitates specialized systems with high-performance processing capabilities, this becomes much more challenging. Class imbalance is a prevalent issue in both traditional and Big Data classification that needs to be handled. Machine learning algorithms may be biased in predicting the majority class in Big Data if there is a significant class gap between the majority and minority classes. When it comes to data processing and analysis in Apache Spark, imbalanced data can lead to several issues. It is critical to use sampling approaches to handle uneven data to minimize these problems. In this work, the degree of data imbalance is compared using logistic regression algorithms in Apache Spark framework. The experimental findings demonstrate the importance of data imbalance and how well it can affect the stability and performance of the Apache Spark application.