This exploratory paper builds on the EMMSAD 2024 paper “Could a Large Language Model Contribute Significantly to Requirements Analysis?” Eight versions of each of three LLM prompts (for system structure, analysis, and recommendations) were applied to three 3000+ word case studies. Those versions expressed different “treatments” including a control with no RAG augmentation, a version with RAG augmentation based on an analysis template used by MBA and EMBA students, and six other versions based on theoretical approaches such as activity theory, a BPM design space, work system principles, and so on. The LLM responses were somewhat reliable for summarizing system structure, less reliable for summarizing an analysis, and often generic and impractical for recommendations because the LLM did not understand contexts. This new paper adds three new capabilities: 1) RAG augmentation using a knowledge base consisting of “knowledge objects” built on work system theory, 2) application of that knowledge base using chain-of-thought reasoning, 3) inclusion of direct feedback from analysts during an analysis process in order to correct errors and to extend the prompt in new directions. Examples are used to illustrate results from applying those capabilities to 3 disparate case studies.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

AI-Based Requirements Analysis Assistant that Applies Explicit Knowledge and Includes Humans in the Loop

  • Steven Alter

摘要

This exploratory paper builds on the EMMSAD 2024 paper “Could a Large Language Model Contribute Significantly to Requirements Analysis?” Eight versions of each of three LLM prompts (for system structure, analysis, and recommendations) were applied to three 3000+ word case studies. Those versions expressed different “treatments” including a control with no RAG augmentation, a version with RAG augmentation based on an analysis template used by MBA and EMBA students, and six other versions based on theoretical approaches such as activity theory, a BPM design space, work system principles, and so on. The LLM responses were somewhat reliable for summarizing system structure, less reliable for summarizing an analysis, and often generic and impractical for recommendations because the LLM did not understand contexts. This new paper adds three new capabilities: 1) RAG augmentation using a knowledge base consisting of “knowledge objects” built on work system theory, 2) application of that knowledge base using chain-of-thought reasoning, 3) inclusion of direct feedback from analysts during an analysis process in order to correct errors and to extend the prompt in new directions. Examples are used to illustrate results from applying those capabilities to 3 disparate case studies.