This paper investigates the application of association rule mining in the context of big data analytics, with a focus on two widely utilized algorithms, Apriori and PCY. Using a simulated dataset of consumer electronics purchases, we rigorously evaluate the efficiency, memory consumption, and scalability of these algorithms in the identification of frequent itemsets and the extraction of meaningful association rules. The straightforward design of the Apriori algorithm is compared with the optimized approach of the PCY algorithm, which employs hash-based counting and bitmap filtering to enhance computational performance. Through detailed implementations in Python and Apache Spark, we conduct a comprehensive experimental analysis, shedding light on the relative strengths and limitations of each algorithm. The results demonstrate PCY’s superior scalability for large datasets, while Apriori’s simplicity remains advantageous for smaller-scale applications. This research provides valuable insights for data scientists and industry practitioners, aiding them in selecting the most suitable algorithm for real-world big data challenges. By merging theoretical analysis with practical implementation, the paper offers a significant contribution to the advancement of efficient association rule mining techniques in the age of big data.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Unlocking Hidden Patterns: A Comparative Analysis of Apriori and Adaptive PCY Algorithms in Big Data Association Rule Mining for Business Intelligence

  • T. S. Ajai Krishna,
  • S. J. Ramanan,
  • Jakkamsetti Ganesh,
  • S Vijayalakshmi,
  • B Shalini

摘要

This paper investigates the application of association rule mining in the context of big data analytics, with a focus on two widely utilized algorithms, Apriori and PCY. Using a simulated dataset of consumer electronics purchases, we rigorously evaluate the efficiency, memory consumption, and scalability of these algorithms in the identification of frequent itemsets and the extraction of meaningful association rules. The straightforward design of the Apriori algorithm is compared with the optimized approach of the PCY algorithm, which employs hash-based counting and bitmap filtering to enhance computational performance. Through detailed implementations in Python and Apache Spark, we conduct a comprehensive experimental analysis, shedding light on the relative strengths and limitations of each algorithm. The results demonstrate PCY’s superior scalability for large datasets, while Apriori’s simplicity remains advantageous for smaller-scale applications. This research provides valuable insights for data scientists and industry practitioners, aiding them in selecting the most suitable algorithm for real-world big data challenges. By merging theoretical analysis with practical implementation, the paper offers a significant contribution to the advancement of efficient association rule mining techniques in the age of big data.