On Implementing Case-Based Reasoning with Large Language Models

Systems based on Large Language Models (LLMs), such as ChatGPT, have impressive performance but also well-known issues with erroneous output. Retrieval Augmented Generation (RAG), which typically presents the LLM with text snippets of additional knowledge retrieved from an external knowledge base, is a popular method for increasing LLM accuracy. This paper presents initial studies exploring augmenting LLMs with cases rather than snippets and prompting LLMs towards performing case-based reasoning. The studies consider four possible scenarios, exploring the potential benefit of LLMs performing different subparts of the CBR process: (1) a scenario in which the LLM is prompted to adapt a presented case, (2) a scenario in which the LLM is first prompted to perform similarity assessment to select a case from a set of candidates, and then to adapt the selected case, (3) a scenario in which the LLM is prompted to select the two most similar cases to a problem and generate an adapted/combined solution in light of both, and (4) a scenario in which the LLM selects the nearest neighbor and nearest unlike neighbor and generates an adapted/combined solution based on both. Results of tests using Llama and ChatGPT are encouraging for the accuracy benefits of providing LLMs with cases and raise questions for future study.