The prevailing belief in the data and analytics community is that high-quality data is the cornerstone of robust artificial intelligence (AI) models and effective analytics. However, ensuring data quality often requires substantial human effort, which is not only time-consuming and expensive but also often tedious and demotivating. To address this challenge, this study demonstrates that AI can itself be leveraged to intelligently detect low-quality or erroneous data, thereby automating and enhancing data quality control in a heterogeneous and correlated dataset (geospatial location data as a representative example). By integrating both unsupervised and supervised machine learning (ML) techniques, the proposed AI framework aims to improve accuracy and reduce manual efforts. Traditional anomaly detection approaches often falter when applied to datasets containing both spatial and categorical variables with dependencies. Our framework addresses these limitations by combining clustering techniques, string similarity-based standardization, and partitioned isolation forests to correct and detect anomalies. A supervised post-processing feedback step further refines the results, significantly reducing false positives and enhancing precision, thereby minimizing manual review efforts. Companies at the forefront of data-driven transformation are increasingly prioritizing intelligent data quality control as a core element of their data governance practices, elevating it to a top-tier data strategy. This innovative solution is designed to support this strategic shift.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Let AI Clean Data Garbage: An Intelligent Quality Control Framework for Heterogeneous Datasets

  • Alessandra Virga,
  • Maximilian Christoph Kohnen,
  • Ameer Hamza Siraj,
  • Georgia Klapsa,
  • Veselin Petrov,
  • Haonan Wu

摘要

The prevailing belief in the data and analytics community is that high-quality data is the cornerstone of robust artificial intelligence (AI) models and effective analytics. However, ensuring data quality often requires substantial human effort, which is not only time-consuming and expensive but also often tedious and demotivating. To address this challenge, this study demonstrates that AI can itself be leveraged to intelligently detect low-quality or erroneous data, thereby automating and enhancing data quality control in a heterogeneous and correlated dataset (geospatial location data as a representative example). By integrating both unsupervised and supervised machine learning (ML) techniques, the proposed AI framework aims to improve accuracy and reduce manual efforts. Traditional anomaly detection approaches often falter when applied to datasets containing both spatial and categorical variables with dependencies. Our framework addresses these limitations by combining clustering techniques, string similarity-based standardization, and partitioned isolation forests to correct and detect anomalies. A supervised post-processing feedback step further refines the results, significantly reducing false positives and enhancing precision, thereby minimizing manual review efforts. Companies at the forefront of data-driven transformation are increasingly prioritizing intelligent data quality control as a core element of their data governance practices, elevating it to a top-tier data strategy. This innovative solution is designed to support this strategic shift.