Optimizing Blockchain ETL Algorithms: Improvement Strategies Based on RPC Interfaces
摘要
Blockchain technology is widely applied, yet existing data-acquisition methods, such as website API, full nodes, RPC, face limitations and struggle with exponential data growth. To address the challenges of obtaining blockchain transaction data, this paper proposes an optimized RPC-based ETL tool. The proposed method uses request-level parallel invocation, introduces synchronization semaphores, and incorporates path-compression techniques. A request-tracking tree is constructed, which generates SHA256-hashed fingerprints to distinguish node types. A union-find structure compresses its paths, reducing height and making the time complexity of the new request-record update O(1), improving the efficiency of the search and reducing the use of memory. Synchronization semaphores indicate incomplete requests, controlling execution order, avoiding resource contention, and ensuring accurate data processing. Experiments with Ethereum data show that it prevents data omission in collection, boosts efficiency, and enhances the reliability of blockchain data acquisition.