In university systems, traditional methods of information retrieval are often found to be inefficient, leading to frustration among students and staff. This paper presents the development and evaluation of a university-specific chatbot that employs the Retrieval-Augmented Generation (RAG) approach to improve the accuracy and relevance of its responses. Unlike conventional chatbots that depend on intent classification and pre-designed system responses and conversation flows, the proposed chatbot integrates Large Language Models (LLMs) with local university data, enhancing its ability to handle complex queries with context-aware responses and dynamically generated conversation flows. The system architecture includes components such as LangChain for orchestration, a vector store for embedding external knowledge, and a user interface developed using Streamlit. Evaluation results demonstrate that the RAG-based chatbot substantially outperforms traditional LLMs, including GPT-3.5, GPT-4 mini, and GPT-4, in terms of answer accuracy and reliability. In this paper we also reflect on the lessons learned during the chatbot’s development and deployment in a real-world university setting.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Development and Evaluation of a University Chatbot Using Deep Learning: A RAG-Based Approach

  • Kabir Olawore,
  • Michael McTear,
  • Yaxin Bi

摘要

In university systems, traditional methods of information retrieval are often found to be inefficient, leading to frustration among students and staff. This paper presents the development and evaluation of a university-specific chatbot that employs the Retrieval-Augmented Generation (RAG) approach to improve the accuracy and relevance of its responses. Unlike conventional chatbots that depend on intent classification and pre-designed system responses and conversation flows, the proposed chatbot integrates Large Language Models (LLMs) with local university data, enhancing its ability to handle complex queries with context-aware responses and dynamically generated conversation flows. The system architecture includes components such as LangChain for orchestration, a vector store for embedding external knowledge, and a user interface developed using Streamlit. Evaluation results demonstrate that the RAG-based chatbot substantially outperforms traditional LLMs, including GPT-3.5, GPT-4 mini, and GPT-4, in terms of answer accuracy and reliability. In this paper we also reflect on the lessons learned during the chatbot’s development and deployment in a real-world university setting.