A Deduplication and Extraction Algorithm for Frequent Itemsets of Overlapping Data Between Power Categories Based on Variable Time Windows
摘要
In the process of data extraction, the rigid partitioning mechanism of fixed time windows leads to spatiotemporal heterogeneity mismatches in data distribution, resulting in semantic confusion and redundancy accumulation in mining results. To address the issues of overlapping data, redundancy, and duplicate frequent itemsets across power categories, this study proposes a deduplication algorithm for frequent itemsets in overlapping data between power categories based on a variable time window. First, the variable time window length is determined according to the dynamic characteristics of inter-class data and temporal evolution. Then, a topic model is constructed using the variable time window to calculate the distribution probability of frequent itemsets in overlapping data. Subsequently, parallel association rules are applied to mine duplicate frequent itemsets. Finally, the mined frequent itemsets are processed using the dynamic time warping method, and the similarity between itemsets is calculated to extract those exceeding the set threshold, thereby retaining valid itemsets. Test results show that when the variable time window length exceeds 35 s, the distribution probability of frequent itemsets can be effectively calculated. This method exhibits strong mining performance, with a perplexity below 0.13. The deduplication effect is significant, reducing the repetition rate of frequent itemsets from 12.2 to 0.657%. After applying this method, the deduplication rate and the integrity of valid itemsets remain above 92.8% and 90.6%, respectively.