Scholarlink

The design of data enrichment pipelines is a complex task that can be simplified by leveraging Large Language Models (LLMs) for natural language-based creation of Directed Acyclic Graphs (DAGs). This approach aims to democratize data pipeline development by enabling DAG generation through natural language text inputs. Our study explores using LLMs to generate Apache Airflow DAGs. This approach streamlines workflow design and makes complex data enrichment more accessible, while preserving the flexibility and power of Airflow’s ecosystem. We conducted preliminary experiments using the SemT framework to address data enrichment challenges at scale. SemT service model, based on the Explore-Design-Operate methodology, enables flexible data transformation pipelines that work across different environments.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

LLM-Based DAG Creation for Data Enrichment Pipelines in SemT Framework

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Abubakari Alidu,
  • Michele Ciavotta,
  • Flavio De Paoli

摘要

The design of data enrichment pipelines is a complex task that can be simplified by leveraging Large Language Models (LLMs) for natural language-based creation of Directed Acyclic Graphs (DAGs). This approach aims to democratize data pipeline development by enabling DAG generation through natural language text inputs. Our study explores using LLMs to generate Apache Airflow DAGs. This approach streamlines workflow design and makes complex data enrichment more accessible, while preserving the flexibility and power of Airflow’s ecosystem. We conducted preliminary experiments using the SemT framework to address data enrichment challenges at scale. SemT service model, based on the Explore-Design-Operate methodology, enables flexible data transformation pipelines that work across different environments.