Scholarlink

<p>Software languages evolve over time for various reasons, such as the addition of new features. When the language’s grammar definition evolves, textual instances that originally conformed to the grammar become outdated. For DSLs in a model-driven engineering context, there exists a plethora of techniques to co-evolve models with the evolving metamodel. However, these techniques are not geared to support DSLs with a textual grammar—applying them to textual language definitions and instances may lead to the loss of information from the original instances, such as layout information and comments, which are valuable for software comprehension and maintenance. This study systematically evaluates the potential of Large Language Model (LLM)-based solutions in achieving grammar and instance co-evolution for textual DSLs. By applying two advanced language models, Claude Sonnet 4.5 and GPT-5.2, and conducting ten experimental runs per case across ten case languages, we evaluate both the correctness of co-evolved instances and the preservation of human-oriented information such as comments and layout. Our results indicate high performance on small-scale cases (<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\ge \)</EquationSource> <EquationSource Format="MATHML"><math> <mo>≥</mo> </math></EquationSource> </InlineEquation>94% precision and recall for instances with fewer than 20 lines requiring modification), but performance degraded with scale: Claude Sonnet 4.5 maintained 85% recall at 40 lines while GPT-5.2 showed greater sensitivity, failing entirely on the two largest instances. Instance scale also substantially impacts processing efficiency, with Claude’s response time increasing nearly 18-fold for the largest case. In addition, we observe that grammar evolution complexity and deletion granularity impact performance more than change type alone, and prompt transferability across LLMs is limited. These findings identify the conditions under which LLM-based co-evolution succeeds and its scalability limitations, providing practitioners with insights into applicability and researchers with directions for addressing current limitations.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Leveraging LLMs to support co-evolution between definitions and instances of textual DSLs: a systematic evaluation

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月10日

查看全文
  • Weixing Zhang,
  • Bowen Jiang,
  • Yuhong Fu,
  • Anne Koziolek,
  • Regina Hebig,
  • Daniel Strüber

摘要

Software languages evolve over time for various reasons, such as the addition of new features. When the language’s grammar definition evolves, textual instances that originally conformed to the grammar become outdated. For DSLs in a model-driven engineering context, there exists a plethora of techniques to co-evolve models with the evolving metamodel. However, these techniques are not geared to support DSLs with a textual grammar—applying them to textual language definitions and instances may lead to the loss of information from the original instances, such as layout information and comments, which are valuable for software comprehension and maintenance. This study systematically evaluates the potential of Large Language Model (LLM)-based solutions in achieving grammar and instance co-evolution for textual DSLs. By applying two advanced language models, Claude Sonnet 4.5 and GPT-5.2, and conducting ten experimental runs per case across ten case languages, we evaluate both the correctness of co-evolved instances and the preservation of human-oriented information such as comments and layout. Our results indicate high performance on small-scale cases ( \(\ge \) ≥ 94% precision and recall for instances with fewer than 20 lines requiring modification), but performance degraded with scale: Claude Sonnet 4.5 maintained 85% recall at 40 lines while GPT-5.2 showed greater sensitivity, failing entirely on the two largest instances. Instance scale also substantially impacts processing efficiency, with Claude’s response time increasing nearly 18-fold for the largest case. In addition, we observe that grammar evolution complexity and deletion granularity impact performance more than change type alone, and prompt transferability across LLMs is limited. These findings identify the conditions under which LLM-based co-evolution succeeds and its scalability limitations, providing practitioners with insights into applicability and researchers with directions for addressing current limitations.