Similarity Evaluation Between Queries and Chunks in RAG Systems
摘要
While Large Language Models (LLMs) demonstrate excellent performance, their output depends on pre-trained knowledge. It has been pointed out that LLMs have various problems, such as generating outdated or incorrect information and they cannot always generate accurate answers. Retrieval-Augmented Generation (RAG) is a technical means that can mitigate the aforementioned problems by connecting an external database to LLM and retrieving relevant information from it, thereby making it possible to utilize a piece of information not learnt. However, it is difficult to always obtain expected answers even with RAG, and one of the reasons for this is the inability to obtain chunks that contain correct answers to user questions or requests, that is, queries. We consider it important to quantitatively evaluate the effect of structure inside the obtained chunks. In this paper, we focus on the similarity between queries and chunks in RAG architecture. The experiments use three embedding models for validation, and the results show that some models tend to increase the cosine similarity when the relevant words of the query are located at the beginning of the chunks. On the other hand, no such tendency was observed when the related words were placed near the end of the chunk, suggesting that the location of the related words influences the accuracy of acquiring chunks.