RDC-Repair: Towards a Relevance-Driven Approach for Data and Constraints Repair

Over the years data cleaning solutions have focused on modifying data in order to satisfy a set of integrity constraints (ICs). However, constraints may also be erroneous. Consequently, when dealing with an inconsistency, it becomes difficult to determine whether the data is dirty, the ICs or both. In this paper, we address three main challenges: (1) Uncertainty: Since both data and ICs could be dirty, it is difficult to detect the source of the inconsistency and how to repair it. We propose a new metric to evaluate the relevance degree of ICs (Functional Dependencies (FDs)). (2) ICs Accuracy: when repairing ICs, they could be transformed to overrefined ICs preventing errors detection or oversimplified ICs leading to clean data modification. We propose to extract possible FD repairs from association rules with limiting the number of LHS attributes to avoid overrefined FDs. Oversimplified FDs on the other hand, are avoided by allowing FD repairs only when the candidate is close enough to the FD. (3) Data Accuracy: Most existing solutions repairing both data and constraints, focus more on the simultaneous repair and less on the data repair algorithm, which affects the quality of the data updates. We propose to use CSP-DC which updates data when possible, and handles ambiguous repair cases by formulating the data cleaning problem as a constraint satisfaction problem, allowing all repairs to be consistent. Experiments show that our solution outperforms most renowned state-of-the-art approaches and provides accurate constraints and data repairs in a linear time.