Testing the use of a large language model (LLM) for performing data quality assessment

The purpose of this study is to evaluate the capability of large language models (LLMs) to perform data quality assessment on background data with respect to a study scenario.

LLMs generate coherent and contextually relevant text in response to prompts. Using a chat interface and prompting the model in a conversational style, OpenAI’s DaVinci model was prompted to perform a data quality assessment of background data against a study scenario using a revised Pedigree Matrix. The model performed DQA across the temporal, geographic, and technology coverage indicators. The outputs were evaluated for correctness in reasoning as well as the final scoring. Prompts underwent several iterations in some cases to improve the correctness.

The model was able to provide correct reasoning and scores for 100% of scenarios across the temporal coverage, 91% for geographic coverage, and 73% for technology coverage. The technology coverage test was repeated for two scenarios where contextual clues in the prompt were obscured to evaluate if the model was indexing its training data to produce scores. Both additional technology coverage scenarios had 100% success in reasoning and scoring. Basic iteration and refinement of the prompts was successful in improving the correctness of outputs.

The results of this research indicate that LLMs are capable of performing data quality assessment across the temporal, geographic, and technology coverage indicators. Outsourcing DQA to artificial intelligence (A.I.) can eliminate practitioner’s biases and reduce liability for practitioners. The success of an off-the-shelf model and improvement in results from little prompt refinement indicates that a LLM may successfully be fine-tuned for the purpose of performing DQA.