Leveraging Large Language Models for Comprehensive Literature Review: An Exploration of RAND Corporation’s 20-Year Corpus

This research project seeks to address a significant limitation in existing large language model (LLM)-based artificial intelligence systems: their inability to accurately generate literature reviews with precise citations. These AI models, despite their ability to emulate complex patterns of human language and knowledge, often falter in referencing specific sources, leading to inaccurately “hallucinated” citations. Our system tackles this problem for a comprehensive 20-year, 5844-document corpus published by the RAND Corporation. We employ a multi-step process, involving extracting text from the corpus documents, segmenting that text into overlapping chunks, generating LLM embeddings for each chunk, and storing these embeddings in a vector database. User queries generate corresponding embeddings that then facilitate a cosine similarity-based retrieval of the most semantically relevant corpus excerpts along with their associated metadata. The text and metadata retrieved are subsequently summarized and transformed into a comprehensive literature review using the OpenAI API. Preliminary results indicate that this method offers a robust and practical means of generating meaningful literature reviews with accurate citations, with the added benefit of contextual precision. Although the tool’s comprehensive accuracy still requires further assessment, it has demonstrated significant potential as a resource for researchers at the beginning of a project and program directors needing quick overviews of institutional research. This research builds on the existing PaperQA framework, extending its use to a significant real-world corpus. The methodology shows promise for application to other substantial document collections, including non-publicly accessible ones, suggesting its potential utility in various fields such as legal and regulatory industries. As such, this work demonstrates a novel solution to the challenge of literature review and citation generation in large corpora.