Beyond ChatGPT: Benchmarking speech-recognition chatbots for language learning using a novel decision-making framework

Language is an essential component of human communication and interaction. Advances in Artificial Intelligence (AI) technology, specifically in Natural Language Processing (NLP) and speech-recognition, have made is possible for conversational agents, also known as chatbots, to converse with language learners in a way that mimics human speech. Numerous researchers have provided empirical evidence supporting the beneficial impact of voice-based chatbots on language acquisition. As a result, benchmarking Language Learning Chatbot (LLC) for effectiveness is crucial for not only for language learners but also for designers and developers of LLC applications. Three challenges make this process a Multiple Criteria Decision-Making (MCDM) problem, namely multiple language learning features, uncertainty of the feature’s importance level, and data variation. Hence, addressing these issues requires the implementation of an MCDM solution. This research proposes a novel MCMD method to rank LLC applications. Our study extends the Multi Criteria Ranking by Alternative Trace (MCRAT) technique with the Fuzzy Weighted with Zero Inconsistency (FWZIC) approach under the T-Spherical (TS) fuzzy environment. The research methodology is initiated by formulating a decision matrix based on the intersection of LLC applications and nine application features. Subsequently, the TS–FWZIC method is developed to ascertain the weights of the application features. Utilizing these weights and the formulated decision matrix, the MCRAT method is employed to rank the LLC applications. The effectiveness of the proposed method is then assessed through sensitivity analysis and comparative evaluation.