Large language models (LLMs) have been increasingly used to automatically generate BPMN 2.0 process models from natural language process descriptions. Corresponding research aim to reduce modeling efforts and enable domain experts to create sound BPMN 2.0 process models. Prior research has primarily focused on evaluating the feasibility and syntactic quality of LLM-generated process models, whereas their pragmatic quality, and their comprehensibility in particular remain underexplored. This limits existing research, as the benefit and usability of process models depend on how well they can be comprehended by different stakeholders. This paper empirically investigates the comprehensibility of LLM-generated BPMN 2.0 process models by users with different levels of expertise (e.g., novices vs. experts). A controlled eye-tracking experiment is presented during which both novices and experts analyzed five BPMN 2.0 process models we generated with BPMNGen, an LLM-based chatbot. Process model comprehension was assessed with comprehension questions, subjective cognitive load measures, and eye-tracking metrics. The results show that both novices and experts achieve comparable comprehension scores when interpreting LLM-generated process models; however, no statistically significant differences between the groups were observed, and these findings should be interpreted with caution given the limited statistical power of the study. With this study, we provide initial empirical insights into the accessibility of LLM-based process modeling approaches in heterogeneous organizational contexts.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Comprehending LLM-Generated BPMN 2.0 Process Models: Novice vs. Expert Perspectives

  • Maximilian Möller,
  • Luca Franziska Hörner,
  • Manfred Reichert

摘要

Large language models (LLMs) have been increasingly used to automatically generate BPMN 2.0 process models from natural language process descriptions. Corresponding research aim to reduce modeling efforts and enable domain experts to create sound BPMN 2.0 process models. Prior research has primarily focused on evaluating the feasibility and syntactic quality of LLM-generated process models, whereas their pragmatic quality, and their comprehensibility in particular remain underexplored. This limits existing research, as the benefit and usability of process models depend on how well they can be comprehended by different stakeholders. This paper empirically investigates the comprehensibility of LLM-generated BPMN 2.0 process models by users with different levels of expertise (e.g., novices vs. experts). A controlled eye-tracking experiment is presented during which both novices and experts analyzed five BPMN 2.0 process models we generated with BPMNGen, an LLM-based chatbot. Process model comprehension was assessed with comprehension questions, subjective cognitive load measures, and eye-tracking metrics. The results show that both novices and experts achieve comparable comprehension scores when interpreting LLM-generated process models; however, no statistically significant differences between the groups were observed, and these findings should be interpreted with caution given the limited statistical power of the study. With this study, we provide initial empirical insights into the accessibility of LLM-based process modeling approaches in heterogeneous organizational contexts.