This paper introduces an AI-driven approach for automated data quality assessment, scoring, and correction in relational databases, leveraging the power of Large Language Models (LLMs). Traditional data quality checks often rely on syntactic rules, overlooking the semantic context for accurate evaluation. Our method employs LLMs to perform semantic analysis of data, enabling a deeper understanding of data integrity and consistency beyond structural compliance. By analyzing data content and metadata, the LLM generates a comprehensive data quality score, reflecting the semantic accuracy and completeness of the database. Furthermore, the system provides actionable correction suggestions to address identified data quality issues. This scoring and correction system empowers data governance professionals to efficiently identify, prioritize, and remediate data quality issues. We demonstrate the efficacy of our approach through experiments on real-world relational databases, showcasing the ability of LLMs to detect subtle semantic anomalies and provide practical solutions for improved data quality management.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

AI-Driven Semantic Data Quality Assessment and Scoring for Relational Databases

  • Antony Seabra,
  • Claudio Cavalcante,
  • Nicolaas Ruberg,
  • Sergio Lifschitz

摘要

This paper introduces an AI-driven approach for automated data quality assessment, scoring, and correction in relational databases, leveraging the power of Large Language Models (LLMs). Traditional data quality checks often rely on syntactic rules, overlooking the semantic context for accurate evaluation. Our method employs LLMs to perform semantic analysis of data, enabling a deeper understanding of data integrity and consistency beyond structural compliance. By analyzing data content and metadata, the LLM generates a comprehensive data quality score, reflecting the semantic accuracy and completeness of the database. Furthermore, the system provides actionable correction suggestions to address identified data quality issues. This scoring and correction system empowers data governance professionals to efficiently identify, prioritize, and remediate data quality issues. We demonstrate the efficacy of our approach through experiments on real-world relational databases, showcasing the ability of LLMs to detect subtle semantic anomalies and provide practical solutions for improved data quality management.