Scholarlink

Systems engineeringSystems engineering (SE) is evolving with the availability of generative artificial intelligence (AIArtificial Intelligence (AI)) and the demand for a systems-of-systems perspective, formalized under the purview of mission engineeringMission engineering (ME) in the US Department of Defense. Formulating ME problems is challenging because they are open-ended exercises that involve translation of ill-defined problems into well-defined ones that are amenable for engineering development. It remains to be seen to which extent AIArtificial Intelligence (AI) could assist problem formulation objectives. To that end, this paper explores the quality and consistency of multi-purpose Large Language Models (LLMLarge Language Models (LLM)) in supporting ME problem formulation tasks, specifically focusing on stakeholder identification. We identify a relevant reference problem, a NASA space mission designMission design challenge, and document ChatGPT-3.5’s ability to perform stakeholder identification tasks. We execute multiple parallel attempts and qualitatively evaluate LLMLarge Language Models (LLM) outputs, focusing on both their quality and variability. Our findings portray a nuanced picture. We find that the LLMLarge Language Models (LLM) performs well in identifying human-focused stakeholders but poorly in recognizing external systems and environmental factors, despite explicit efforts to account for these. Additionally, LLMsLarge Language Models (LLM) struggle with preserving the desired level of abstraction and exhibit a tendency to produce solution-specific outputs that are inappropriate for problem formulationProblem formulation.. More importantly, we document great variability among parallel threads, highlighting that LLMLarge Language Models (LLM) outputs should be used with caution, ideally by adopting a stochastic view of their abilities. Overall, our findings suggest that, while ChatGPT could reduce some expert workload, its lack of consistency and domain understanding may limit its reliability for problem formulation tasks.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

An Empirical Exploration of ChatGPT’s Ability to Support Problem Formulation Tasks for Mission Engineering and a Documentation of Its Performance Variability

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Max Ofsa,
  • Taylan G. Topcu

摘要

Systems engineeringSystems engineering (SE) is evolving with the availability of generative artificial intelligence (AIArtificial Intelligence (AI)) and the demand for a systems-of-systems perspective, formalized under the purview of mission engineeringMission engineering (ME) in the US Department of Defense. Formulating ME problems is challenging because they are open-ended exercises that involve translation of ill-defined problems into well-defined ones that are amenable for engineering development. It remains to be seen to which extent AIArtificial Intelligence (AI) could assist problem formulation objectives. To that end, this paper explores the quality and consistency of multi-purpose Large Language Models (LLMLarge Language Models (LLM)) in supporting ME problem formulation tasks, specifically focusing on stakeholder identification. We identify a relevant reference problem, a NASA space mission designMission design challenge, and document ChatGPT-3.5’s ability to perform stakeholder identification tasks. We execute multiple parallel attempts and qualitatively evaluate LLMLarge Language Models (LLM) outputs, focusing on both their quality and variability. Our findings portray a nuanced picture. We find that the LLMLarge Language Models (LLM) performs well in identifying human-focused stakeholders but poorly in recognizing external systems and environmental factors, despite explicit efforts to account for these. Additionally, LLMsLarge Language Models (LLM) struggle with preserving the desired level of abstraction and exhibit a tendency to produce solution-specific outputs that are inappropriate for problem formulationProblem formulation.. More importantly, we document great variability among parallel threads, highlighting that LLMLarge Language Models (LLM) outputs should be used with caution, ideally by adopting a stochastic view of their abilities. Overall, our findings suggest that, while ChatGPT could reduce some expert workload, its lack of consistency and domain understanding may limit its reliability for problem formulation tasks.