Layered Data-Centric AI to Streamline Data Quality Practices for Enhanced Automation
摘要
Most artificial intelligence (AI) applications are designed under the model-centric AI (MCAI) approach, where data scientists aim to optimize the machine learning (ML) models starting with fixed, preprocessed data. However, businesses often struggle with limited datasets, changes in data over time, and limited ML knowledge, making it difficult to maintain data quality. One potential approach is data-centric AI (DCAI), which systematically improves the data quality used to build AI systems. However, rapid growth in data volume leads to challenges in selecting the most suitable operations for enhancing data quality thereby maintaining data accuracy, completeness, consistency, and reliability in real-world applications. To address this gap, we propose a novel framework, namely Layered Data-Centric AI (LDCAI) by expanding upon the existing DCAI Pipeline. LDCAI employs a three-layered approach, starting with the DCAI layer which generates the initial training data. This data is then analyzed and refined in the Data Analysis (DA) layer by a team of data scientists and domain experts. The refined data is later sent to the Data Quality Control (DQC) layer for additional quality checks and improvements by data scientists. The entire process can be iterated as needed to continuously improve data quality, which can be fed back into the DCAI layer’s model to achieve better predictions and outcomes. The implementation is illustrated through a scalable and customizable architecture designed for optimizing data quality within AI systems.