The exponential growth of data in the era of big data and the Internet of Things (IoT) has made traditional centralized association rule mining algorithms inefficient. These algorithms encounter several challenges, including bandwidth limitations, energy constraints, low processing power, privacy concerns, insufficient storage capacity, and limited scalability. This led to the development of Distributed Association Rule Mining (D-ARM). This study implements the FP-Growth algorithm in a parallel computing environment using Apache Spark. The proposed approach enhances scalability and computational efficiency by leveraging Spark’s distributed processing capabilities. The performance of the distributed FP-Growth algorithm is evaluated and compared with the traditional centralized approach using different datasets. Experimental results demonstrate that the Spark-based implementation significantly reduces execution time and improves efficiency, making it suitable for large-scale data mining tasks. This study highlights the advantages of distributed computing in mining association rules and provides insights into the optimization of parallel data mining techniques.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Efficient Mining of Association Rules Using Apache Spark

  • Bhaswati Baishya,
  • Bhabesh Nath

摘要

The exponential growth of data in the era of big data and the Internet of Things (IoT) has made traditional centralized association rule mining algorithms inefficient. These algorithms encounter several challenges, including bandwidth limitations, energy constraints, low processing power, privacy concerns, insufficient storage capacity, and limited scalability. This led to the development of Distributed Association Rule Mining (D-ARM). This study implements the FP-Growth algorithm in a parallel computing environment using Apache Spark. The proposed approach enhances scalability and computational efficiency by leveraging Spark’s distributed processing capabilities. The performance of the distributed FP-Growth algorithm is evaluated and compared with the traditional centralized approach using different datasets. Experimental results demonstrate that the Spark-based implementation significantly reduces execution time and improves efficiency, making it suitable for large-scale data mining tasks. This study highlights the advantages of distributed computing in mining association rules and provides insights into the optimization of parallel data mining techniques.