Software requirements analysis is increasingly automated by applying natural language processing (NLP) tools, enhancing efficiency and precision. This research employs the Mendeley FR_NFR dataset to evaluate the classification of functional requirements (FR) and non-functional requirements (NFR) utilising three NLP tools: NLTK, OpenAI, and spaCy. The evaluation uses performance indicators like F1-score, recall, accuracy, precision, and confusion matrices. OpenAI is a good option for high-stakes applications because of its 94% F1 score and exceptional accuracy, even with the related API expenses. With 83% accuracy and 0.1 s per query, SpaCy is ideal for real-time applications because it balances speed and efficiency. With its 68% accuracy rate, NLTK’s rule-based methodology is still a viable choice for prototyping or in controlled settings where transparency is crucial. With an average accuracy of 92%, the results show that OpenAI’s transformer-based model performs better than NLTK and spaCy, even though spaCy has an advantage in entity recognition. This study provides practitioners with critical insights by elucidating the trade-offs between accuracy, interpretability, and computational efficiency.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluating NLP Tools for AI in Software Requirements Analysis

  • Cornelius Chimuanya Okechukwu,
  • Radek Silhavy,
  • Petr Silhavy

摘要

Software requirements analysis is increasingly automated by applying natural language processing (NLP) tools, enhancing efficiency and precision. This research employs the Mendeley FR_NFR dataset to evaluate the classification of functional requirements (FR) and non-functional requirements (NFR) utilising three NLP tools: NLTK, OpenAI, and spaCy. The evaluation uses performance indicators like F1-score, recall, accuracy, precision, and confusion matrices. OpenAI is a good option for high-stakes applications because of its 94% F1 score and exceptional accuracy, even with the related API expenses. With 83% accuracy and 0.1 s per query, SpaCy is ideal for real-time applications because it balances speed and efficiency. With its 68% accuracy rate, NLTK’s rule-based methodology is still a viable choice for prototyping or in controlled settings where transparency is crucial. With an average accuracy of 92%, the results show that OpenAI’s transformer-based model performs better than NLTK and spaCy, even though spaCy has an advantage in entity recognition. This study provides practitioners with critical insights by elucidating the trade-offs between accuracy, interpretability, and computational efficiency.