Research and Application of an Incremental Extraction Scheme for Monitoring Power Equipment Data

The monitoring data of power equipment exhibit characteristics of multiple sources, a large amount of data, and high real-time performance. To effectively extract information from these data, it is imperative to extract monitoring data from various equipment and subject it to a series of processing steps. Data extraction is a crucial stage in the ETL (extraction, transformation, loading) process. In this paper, we conduct a comprehensive analysis of various incremental extraction capture mechanisms, highlighting their respective advantages and disadvantages and applicable conditions. We then select CDC technology, which involves analysing the database’s log to identify changed data and using the incremental extraction method to extract the monitoring data of power equipment. Our analysis and experimental results demonstrate that the proposed method ensures the accuracy and integrity of data to a significant extent compared to other incremental extraction mechanisms. Furthermore, it addresses the incremental extraction challenges between databases in the power equipment monitoring environment and improves the efficiency and performance of data extraction.