In agile software development, user stories (US) and acceptance criteria (AC) are popular ways of recording requirements. While guidelines have been proposed in the literature to assess the quality of US and AC, their correct application remains a manual task. In this work, we designed both a machine learning (ML) and a natural language processing (NLP) classifier for automatically assessing agile software requirements following quality indicators found in the literature. We evaluated their performance to improve the quality of requirements in a user study, as well as the users’ perceptions on the usage of such tools as writing aids. While improvements were notable in the quality of requirements written by participants, the improvements were more marginal when using the NLP classifier compared to the ML one. However, participants reported more satisfaction towards the NLP classifier for its “explainability” compared to the ML one.