The increasing advent of approaches that adopt Large Language Models for text-to-model generation is accompanied by a variety of experimental designs for their evaluation. Inspired by “The Prompt Report”, which provides a systematic account of prompting techniques for prompt engineering, we present a structured synthesis of experimental designs used to evaluate LLM-based text-to-model approaches. Through a systematic literature review, we compile a compendium of experimental designs, organized into six tree-structured categories. This compendium integrates both experimental design principles from traditional empirical research and challenges unique to LLM usage. For example, specific to LLMs, we identify studies that conduct an ablation study to isolate the effect of individual prompting techniques, as well as those that address cascading errors introduced by intermediate LLM outputs. The compendium is intended to inform the design of future evaluations in this emerging field, offering researchers structured support and guidance.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

The Interim Experiment Report: A Systematic Account of The Experimental Design of Large Language Model-Based Text-to-Model Approaches

  • Sybren de Kinderen,
  • Qin Ma,
  • Jonathan Silva Mercado,
  • Karolin Winter

摘要

The increasing advent of approaches that adopt Large Language Models for text-to-model generation is accompanied by a variety of experimental designs for their evaluation. Inspired by “The Prompt Report”, which provides a systematic account of prompting techniques for prompt engineering, we present a structured synthesis of experimental designs used to evaluate LLM-based text-to-model approaches. Through a systematic literature review, we compile a compendium of experimental designs, organized into six tree-structured categories. This compendium integrates both experimental design principles from traditional empirical research and challenges unique to LLM usage. For example, specific to LLMs, we identify studies that conduct an ablation study to isolate the effect of individual prompting techniques, as well as those that address cascading errors introduced by intermediate LLM outputs. The compendium is intended to inform the design of future evaluations in this emerging field, offering researchers structured support and guidance.