Scholarlink

<p>The generation of Business Process Model and Notation (BPMN) XML outputs from textual process descriptions presents a promising application for large language models (LLMs), yet it introduces significant challenges due to the structured and precise nature of process modeling. This study evaluates the performance of LLMs—Mistral, GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet—in BPMN generation, employing prompt engineering strategies across Simple, Medium, and Complex process descriptions to establish a baseline. Our findings reveal key limitations in LLMs, including limited output control, input presentation dependencies, and a lack of explainability, particularly for complex processes with nested flows and intricate dependencies. To address these challenges, we propose a novel Description-to-DOT pipeline utilizing a fine-tuned Qwen2.5 14B Coder model, trained on the MaD dataset of process description-DOT representation pairs. The novelty of the Description-to-DOT pipeline lies in its use of Graphviz DOT format as an intermediate representation, which requires generating fewer tokens and enables faster completion, followed by a Python script that converts DOT to BPMN XML in milliseconds—a significant efficiency improvement over the direct Description-to-BPMN pipeline, with the Description-to-DOT pipeline being approximately 6 times faster for Medium processes and 11 times faster for Complex processes. Experimental results demonstrate that the fine-tuned model significantly outperforms the evaluated LLMs, achieving accurate BPMN generation across all complexity levels. This study contributes: (1) Identification of LLM limitations in BPMN generation, such as logical inconsistencies, (2) A novel Description-to-DOT pipeline enhancing efficiency and accuracy, (3) A new benchmark dataset from the MaD dataset for Description-to-BPMN tasks, and (4) Comprehensive validation of the approach across complexity levels. These findings demonstrate the transformative potential of fine-tuned SLMs, with the Qwen2.5 Coder 14B enabling a scalable Description-to-DOT pipeline that excels in BPMN automation across complexity levels, validated on the MaD dataset.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Size matters less: how fine-tuned small LLMs excel in BPMN generation

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年11月20日

查看全文
  • Ghada Soliman,
  • Nour Abdel Wars,
  • Hagar Hesham

摘要

The generation of Business Process Model and Notation (BPMN) XML outputs from textual process descriptions presents a promising application for large language models (LLMs), yet it introduces significant challenges due to the structured and precise nature of process modeling. This study evaluates the performance of LLMs—Mistral, GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet—in BPMN generation, employing prompt engineering strategies across Simple, Medium, and Complex process descriptions to establish a baseline. Our findings reveal key limitations in LLMs, including limited output control, input presentation dependencies, and a lack of explainability, particularly for complex processes with nested flows and intricate dependencies. To address these challenges, we propose a novel Description-to-DOT pipeline utilizing a fine-tuned Qwen2.5 14B Coder model, trained on the MaD dataset of process description-DOT representation pairs. The novelty of the Description-to-DOT pipeline lies in its use of Graphviz DOT format as an intermediate representation, which requires generating fewer tokens and enables faster completion, followed by a Python script that converts DOT to BPMN XML in milliseconds—a significant efficiency improvement over the direct Description-to-BPMN pipeline, with the Description-to-DOT pipeline being approximately 6 times faster for Medium processes and 11 times faster for Complex processes. Experimental results demonstrate that the fine-tuned model significantly outperforms the evaluated LLMs, achieving accurate BPMN generation across all complexity levels. This study contributes: (1) Identification of LLM limitations in BPMN generation, such as logical inconsistencies, (2) A novel Description-to-DOT pipeline enhancing efficiency and accuracy, (3) A new benchmark dataset from the MaD dataset for Description-to-BPMN tasks, and (4) Comprehensive validation of the approach across complexity levels. These findings demonstrate the transformative potential of fine-tuned SLMs, with the Qwen2.5 Coder 14B enabling a scalable Description-to-DOT pipeline that excels in BPMN automation across complexity levels, validated on the MaD dataset.