Development of RAG System for Digital Transformation of Local Government and Considering Optimal Document-Segmentation Methods

Administrative documents are accumulating yearly, increasing the burden of document retrieval in administrative work. We developed a retrieval augmented generation (RAG) system for administrative digital transformation and tested its effectiveness. We presented participants with tasks related to administrative issues and compared the accuracy and time taken to respond using the RAG system with those using traditional manual systems. We also set up groups with varying levels of experience and knowledge, i.e., administrative officers in charge, officers from other departments, and students, to analyze the effectiveness of the RAG system in each group. For short-answer questions, the administrative officers in charge achieved a 93.3% correct-answer rate, comparable to manual work, while reducing working time by 23.6%. Students improved their correct-answer rate by 40.0% and reduced working time by 79.8% through the RAG system use. For descriptive questions, however, the system use led to lower evaluations. The evaluation for descriptive questions was based on accuracy of the response, completeness of the response, accuracy of the background, context, and reasons, and completeness of the background, context, and reasons. Retrieval accuracy increased with longer segments, with longer segments yielding higher correct-answer rate in short-answer questions and shorter segments resulted in higher evaluations for descriptive questions. We also explored the optimal document-segmentation character count for the RAG system in administration tasks.