Retrieval Augmented Generation with LLMs for Explaining Business Process Models

Large language models (LLMs) and retrieval augmented generation (RAG) are undergoing rapid development. Considering a case base as a memory in a RAG system provides novel opportunities for text generation. In this paper, we investigate the role Case-Based Reasoning (CBR) could play for supporting RAG systems in generating accessible explanations of business process models. We experiment with two different case bases in a RAG system. Case base a) is dedicated to support prompt chaining by reusing index knowledge on the cases with the aim to deal with large process models that do not fit into the context window size of a recent LLM. Second, case base b) contains model-text pairs to serve as in-context examples to enhance prompt templates. Approach b) aims to improve the quality of generated text explanations for process models of normal size. Our contribution opens a novel application area for process-oriented CBR. Further, our case-based RAG system provides a contemporary alternative to traditional Natural Language Processing pipelines. The experimental results contribute to gain some insights on an inherent capability threshold of GPT-4 at which the performance decreases much earlier than having reached the given context window size, on the number of retrieved cases a recent RAG system should use as in-context examples, and on suitable prompt templates.