Scholarlink

The escalating complexity of modern data landscapes necessitates innovative approaches to streamline Extract, Transform, Load (ETL) processes. This research investigates the potential of Large Language Models (LLMs), specifically BERT and T5, to automate metadata extraction from tabular data exclusively based on data values, bypassing the need for traditional header information. Our study evaluates the ability of pre-trained LLMs to accurately predict schema elements such as data types, constraints, formats, and relationships with minimal training data. Using the diverse AdventureWorks datasets, we rigorously test the LLMs capacity to handle real-world ETL challenges. Our findings demonstrate that LLMs can effectively predict metadata with high accuracy, significantly reducing manual effort and accelerating ETL pipelines. By integrating LLMs into ETL workflows, we achieve substantial improvements in efficiency, timeliness, and accuracy. This research underscores the transformative potential of LLMs in revolutionizing data engineering practices and provides a foundation for future explorations in this domain.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

An Evaluation of Data Pipelines with Large Language Models

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Jayachander Surbiryala,
  • Subhashree Bal,
  • Antorweep Chakravorty

摘要

The escalating complexity of modern data landscapes necessitates innovative approaches to streamline Extract, Transform, Load (ETL) processes. This research investigates the potential of Large Language Models (LLMs), specifically BERT and T5, to automate metadata extraction from tabular data exclusively based on data values, bypassing the need for traditional header information. Our study evaluates the ability of pre-trained LLMs to accurately predict schema elements such as data types, constraints, formats, and relationships with minimal training data. Using the diverse AdventureWorks datasets, we rigorously test the LLMs capacity to handle real-world ETL challenges. Our findings demonstrate that LLMs can effectively predict metadata with high accuracy, significantly reducing manual effort and accelerating ETL pipelines. By integrating LLMs into ETL workflows, we achieve substantial improvements in efficiency, timeliness, and accuracy. This research underscores the transformative potential of LLMs in revolutionizing data engineering practices and provides a foundation for future explorations in this domain.