This study empirically assesses how effectively two Large Language Models (LLMs), <Emphasis FontCategory="NonProportional">OpenAI o3-mini</Emphasis> and <Emphasis FontCategory="NonProportional">DeepSeek R1</Emphasis>, can transform terse feature-request titles from open-source software (OSS) issue trackers into well-formed software requirements. It further examines how prompt-engineering strategies shape requirement quality and evaluates a scalable “LLM-as-a-Judge” approach for automated quality assessment based on three ISO/IEC/IEEE 29148:2018 quality attributes: Unambiguity, Verifiability, and Singularity. We extract 150 feature-request titles from five OSS repositories and pair each title with every combination of two LLMs and three prompt styles, producing 900 candidate requirements. An independent evaluator LLM rates each requirement on these three quality attributes using a Likert scale and provides textual rationales. Ordinal data are analyzed with descriptive statistics and non-parametric tests, complemented by thematic analysis of the evaluator’s explanations. A targeted human validation study with five evaluators on a stratified sample of 50 requirements assesses the reliability of the automated judge. LLMs often produce high-quality requirements; however, scores vary with input clarity and prompt design. Few-shot prompting consistently boosts Singularity, while Expert Identity prompting sometimes raises Verifiability but frequently harms Singularity. <Emphasis FontCategory="NonProportional">o3-mini</Emphasis> shows a modest but significant edge in Singularity. Qualitative review echoes these patterns, revealing trade-offs between added detail and focus. The LLM-as-a-Judge protocol delivers consistent, scalable evaluations whose scores correlate significantly with aggregate human judgment (<InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(p < 0.05\)</EquationSource> </InlineEquation> for all three attributes). Modern LLMs can expedite the drafting of initial software requirements from informal OSS inputs, but their output quality hinges on careful prompt selection and the inherent clarity of the source title. Prompt effects are model-dependent and may introduce trade-offs among quality attributes, so human oversight remains indispensable for refinement. The LLM-as-a-Judge framework proves a practical, human-validated technique for large-scale evaluation, enabling rapid, reproducible insights into LLM-driven requirements engineering workflows.

From issue titles to requirements: an empirical study of large language models and prompt engineering strategies

期刊论文
出版模式：

开放获取
发表日期：

2026年5月26日

查看全文

Guilherme Paiva,
Edna Dias Canedo,
Geraldo Pereira Rocha Filho

摘要

This study empirically assesses how effectively two Large Language Models (LLMs), OpenAI o3-mini and DeepSeek R1, can transform terse feature-request titles from open-source software (OSS) issue trackers into well-formed software requirements. It further examines how prompt-engineering strategies shape requirement quality and evaluates a scalable “LLM-as-a-Judge” approach for automated quality assessment based on three ISO/IEC/IEEE 29148:2018 quality attributes: Unambiguity, Verifiability, and Singularity. We extract 150 feature-request titles from five OSS repositories and pair each title with every combination of two LLMs and three prompt styles, producing 900 candidate requirements. An independent evaluator LLM rates each requirement on these three quality attributes using a Likert scale and provides textual rationales. Ordinal data are analyzed with descriptive statistics and non-parametric tests, complemented by thematic analysis of the evaluator’s explanations. A targeted human validation study with five evaluators on a stratified sample of 50 requirements assesses the reliability of the automated judge. LLMs often produce high-quality requirements; however, scores vary with input clarity and prompt design. Few-shot prompting consistently boosts Singularity, while Expert Identity prompting sometimes raises Verifiability but frequently harms Singularity. o3-mini shows a modest but significant edge in Singularity. Qualitative review echoes these patterns, revealing trade-offs between added detail and focus. The LLM-as-a-Judge protocol delivers consistent, scalable evaluations whose scores correlate significantly with aggregate human judgment ( \(p < 0.05\) for all three attributes). Modern LLMs can expedite the drafting of initial software requirements from informal OSS inputs, but their output quality hinges on careful prompt selection and the inherent clarity of the source title. Prompt effects are model-dependent and may introduce trade-offs among quality attributes, so human oversight remains indispensable for refinement. The LLM-as-a-Judge framework proves a practical, human-validated technique for large-scale evaluation, enabling rapid, reproducible insights into LLM-driven requirements engineering workflows.