A Multi-level Pruning Method for Medical Redundant Data Based on Data Lake
摘要
As the volume of data in the medical field grows exponentially, the issue of data redundancy has become increasingly prominent. For instance, in the case of the large public medical dataset MIMIC-III, the redundancy rate in hospitalization records reaches as high as 97%–98%. This redundancy not only significantly increases storage costs but also reduces data processing efficiency. To date, there has been no proposed method for table-level redundant data pruning specifically tailored to the unique data types found in medical datasets. To address this issue, this paper introduces a multi-level pruning method based on a data lake, which efficiently eliminates redundancy through a hierarchical processing strategy. The method progressively narrows the search space by constructing an inclusion graph (Containment Graph Construction) combined with the multi-level pruning technique, enabling the efficient identification and removal of redundant data. Additionally, to ensure the efficiency, accuracy, and clinical practicality of the pruning results, this paper proposes a “Clinical Decision Making” mechanism. Experimental results demonstrate that this approach can efficiently process large scale medical data and accurately eliminate approximately 23% of redundant data.