Scholarlink

<p>The design structure matrix (DSM) is an established method for modelling design dependencies but manually putting one together can be resource intensive. The Auto-DSM workflow integrates a large language model (LLM) with retrieval-augmented generation (RAG) to extract system dependencies from input data, which are then used to automatically generate a corresponding DSM. This paper reports on an evaluation study that uses the Auto-DSM workflow as a basis to evaluate the retrieval of asymmetrical direct and indirect system dependencies from text data. Five LLMs, namely GPT-4o, GPT-4, Llama 3, DeepSeek-R1, and TinyLlama were used in this work. Auto-DSM with GPT-4 produced a complete DSM with an accuracy of 0.981 (<i>SD</i> = 0.025, <i>N</i> = 600) when plain dependency descriptions were used and reached a full accuracy of 1.000 (<i>SD</i> = 0.000, <i>N</i> = 5) when the same dependencies were presented in the form of patent claims. It was revealed that the way system entities are named in input data can affect accuracy and the reporting of path distance between entities is influenced by the writing style and format of the data. The findings of this work can be used to support the development of automated DSM generation, enabling more advanced DSM techniques to be built on.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

From text to DSM: evaluating the impact of writing style and entity naming on LLM-based retrieval of asymmetrical indirect design dependencies

  • 期刊论文

  • 发表日期：

    2026年3月6日

查看全文
  • Edwin C. Y. Koh

摘要

The design structure matrix (DSM) is an established method for modelling design dependencies but manually putting one together can be resource intensive. The Auto-DSM workflow integrates a large language model (LLM) with retrieval-augmented generation (RAG) to extract system dependencies from input data, which are then used to automatically generate a corresponding DSM. This paper reports on an evaluation study that uses the Auto-DSM workflow as a basis to evaluate the retrieval of asymmetrical direct and indirect system dependencies from text data. Five LLMs, namely GPT-4o, GPT-4, Llama 3, DeepSeek-R1, and TinyLlama were used in this work. Auto-DSM with GPT-4 produced a complete DSM with an accuracy of 0.981 (SD = 0.025, N = 600) when plain dependency descriptions were used and reached a full accuracy of 1.000 (SD = 0.000, N = 5) when the same dependencies were presented in the form of patent claims. It was revealed that the way system entities are named in input data can affect accuracy and the reporting of path distance between entities is influenced by the writing style and format of the data. The findings of this work can be used to support the development of automated DSM generation, enabling more advanced DSM techniques to be built on.