Context and motivation: Large language models (LLMs) are increasingly used to address requirements engineering (RE) tasks, including trace-link recovery, legal compliance, model generation, and others. Question/problem: Most of the existing studies rely on static, non-adaptive prompting strategies that do not fully harness the models’ capabilities. Specifically, these studies overlook the potential of automatic prompting engineering (APE), a technique that allows LLMs to self-generate and fine-tune prompts to improve task performance. Principal ideas/results: This research preview aims to study the effectiveness of APE techniques in LLM-powered RE tasks. As a preliminary step, we perform a benchmarking study in which we compare APE techniques with more traditional prompting solutions for the task of requirements classification. Our results show that, on average and with some exceptions, APE outperforms the baselines. We outline research avenues, including the evaluation and tailoring of APE for other RE tasks, and considering the human-in-the-loop. Contribution: To the best of our knowledge, this is the first study to introduce APE in RE, paving the way for a deeper exploration of LLMs’ potential in this field.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Automatic Prompt Engineering: The Case of Requirements Classification

  • Mohammad Amin Zadenoori,
  • Liping Zhao,
  • Waad Alhoshan,
  • Alessio Ferrari

摘要

Context and motivation: Large language models (LLMs) are increasingly used to address requirements engineering (RE) tasks, including trace-link recovery, legal compliance, model generation, and others. Question/problem: Most of the existing studies rely on static, non-adaptive prompting strategies that do not fully harness the models’ capabilities. Specifically, these studies overlook the potential of automatic prompting engineering (APE), a technique that allows LLMs to self-generate and fine-tune prompts to improve task performance. Principal ideas/results: This research preview aims to study the effectiveness of APE techniques in LLM-powered RE tasks. As a preliminary step, we perform a benchmarking study in which we compare APE techniques with more traditional prompting solutions for the task of requirements classification. Our results show that, on average and with some exceptions, APE outperforms the baselines. We outline research avenues, including the evaluation and tailoring of APE for other RE tasks, and considering the human-in-the-loop. Contribution: To the best of our knowledge, this is the first study to introduce APE in RE, paving the way for a deeper exploration of LLMs’ potential in this field.