UniBot is a domain-specific chatbot designed to streamline access to internal documents within a university context by integrating information retrieval (IR) techniques with large language models (LLMs). The system addresses a critical need in educational institutions: enabling students and staff to quickly access information from internal documents (such as academic policies, announcements, schedules) through natural language queries. The proposed approach employs a Retrieval-Augmented Generation (RAG) architecture, which combines a vector-based document retrieval pipeline with a LLM to generate accurate, context-informed answers. Theoretical foundations related to chatbots, LLMs, and RAG are presented, followed by a detailed overview of UniBot’s design and implementation. Key components include a document indexing module, a vector database for semantic search, and a user-facing conversational interface. In a pilot deployment, UniBot was evaluated both performance metrics (via the RAGAs framework, including context recall and ROUGE) and user studies involving students and academic staff. The results indicate that integrating IR with LLMs substantially improves the relevance and factuality of chatbot responses, reducing instances of hallucinations. Users reported a high satisfaction level (average 4.12/5) with the system’s ability to answer in-domain questions, though some challenges remain with handling out-of-scope queries and response latency. Overall, our findings demonstrate that IR-augmented LLM chatbots can effectively automate information access in educational settings, alleviating administrative workloads and improving user experience.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Integrating Information Retrieval and LLMs: A Document Retrieval Chatbot in Education Settings

  • Duy Dang Khoa Nguyen,
  • Vi Kiet Mach,
  • Thang Le Dinh,
  • Cuong Pham-Nguyen

摘要

UniBot is a domain-specific chatbot designed to streamline access to internal documents within a university context by integrating information retrieval (IR) techniques with large language models (LLMs). The system addresses a critical need in educational institutions: enabling students and staff to quickly access information from internal documents (such as academic policies, announcements, schedules) through natural language queries. The proposed approach employs a Retrieval-Augmented Generation (RAG) architecture, which combines a vector-based document retrieval pipeline with a LLM to generate accurate, context-informed answers. Theoretical foundations related to chatbots, LLMs, and RAG are presented, followed by a detailed overview of UniBot’s design and implementation. Key components include a document indexing module, a vector database for semantic search, and a user-facing conversational interface. In a pilot deployment, UniBot was evaluated both performance metrics (via the RAGAs framework, including context recall and ROUGE) and user studies involving students and academic staff. The results indicate that integrating IR with LLMs substantially improves the relevance and factuality of chatbot responses, reducing instances of hallucinations. Users reported a high satisfaction level (average 4.12/5) with the system’s ability to answer in-domain questions, though some challenges remain with handling out-of-scope queries and response latency. Overall, our findings demonstrate that IR-augmented LLM chatbots can effectively automate information access in educational settings, alleviating administrative workloads and improving user experience.