A Comparison of Frequent Itemset Mining Algorithms in Distributed File Systems Using PySpark: A-Priori Versus PCY
摘要
Frequent itemset mining and association rule generation are common topics in mining massive datasets or big data, the purpose of which is to discover itemsets bought together with a significant proportion. Traditional algorithms are mostly implemented and evaluated as single-machine in-memory programs that are incapable of large-scale systems. In the study, the authors contrast the performance of two algorithms for frequent itemset mining, including A-Priori and Park-Chen-Yu (PCY), in the Hadoop Distribution File System (HDFS) using the PySpark framework. Empirical results in practical datasets emphasize the remarkable performance of PCY thanks to eliminating non-frequent pairs using buckets of candidate ones.