Incremental mining algorithms for generating and updating frequent patterns for dynamic databases against insert, update, and support changes

Developing algorithms for scalable and efficient itemset mining for large incremental databases is paramount. When there is a significant change in the database or a threshold change in parameters, reinventing the wheel for discovering frequent patterns is the cause of concern. In this regard, the state-of-the-art algorithms are inadequate in providing a complete solution. Another key disadvantage of the existing incremental mining algorithms is that they require huge memory and time to find patterns. High computational costs and a lack of faster convergence are not feasible for real-time domain-specific applications. This paper addresses the problems above by proposing efficient and scalable frequent itemset mining algorithms suitable for large incremental databases. Incremental Frequent Pattern Mining for Insert Problem (IFPM-IP), Incremental Frequent Pattern Mining for Update Problem (IFPM-UP), and Incremental Frequent Pattern Mining for Support Change Problem (IFPM-SCP) are the proposed algorithms to handle incremental mining of frequent itemsets on insert, update and support change operations, respectively. These algorithms exploit a data structure known as nodeset which needs either pre-order code or post-order code of each node, leading to faster convergence and memory efficiency. The proposed algorithms efficiently address the requirement of incremental frequent itemset mining against substantial changes in the database or adjustments to parameter thresholds without necessitating the rebuilding of the POC tree. Besides, the algorithms are designed to run in parallel mode with the support of a cloud-based distributed computing framework, leading to further improvement in efficiency and scalability. Experimental results revealed that the proposed algorithms, such as FP-growth, FUFP-tree, Pre-FUFP, FCFPIM, and FIUFP-growth, outperform state of the art regarding execution time. With 30,000 new records and 3% support, IFPM-IP is 1.8 to 4.5 times faster than the existing methods.