Can Large Language Models Accelerate Digital Transformation by Generating Expert-Like Systems Engineering Artifacts? Insights from an Empirical Exploration

Generative artificial intelligence (AI), such as large language models (LLMs), could dramatically alter the way design information is represented and managed. Systems engineering (SE) literature has been exploring the use of AI; however, these efforts mostly focused on formulation of knowledge databases, virtual assistants, and design evaluators. We contend that LLMs could simultaneously help both SE and digital transformation, by generating design artifacts that could be directly incorporated into digital ecosystems. To that end, this paper provides an existence proof for the ability of LLMs to generate coherent, expert-like segments of SE artifacts in natural language. We illustrate this by creating an empirical probe that leverages existing SE artifacts and various prompt formulation strategies to create LLM-generated SE artifacts. We then use a semantic comparison framework to quantitatively evaluate the closeness of these LLM-generated artifacts to human-expert-generated ones. We find that LLMs can generate expert-like artifacts even with a relatively small training set; however, how LLMs are utilized in terms of the prompting approach has a significant influence on the quality of artifacts. The findings of this study should serve as a conservative estimate of the potential of LLMs to assist with SE tasks, including digital transformation.