RAG Under Load: Analytical and Empirical Modeling of Retrieval–Generation Performance at Scale
摘要
Retrieval-Augmented Generation (RAG) has emerged as the dominant architecture for grounding large language models with factual and context-aware information. Yet, little is known about its end-to-end performance behavior under load—particularly how retriever latency, token generation rate, and concurrency interact to determine overall throughput and user-perceived responsiveness. This study presents the first unified framework to our knowledge for analytical and empirical modeling of RAG pipelines under variable load conditions. This study develops a queue-coupled (queuing analysis–based) performance model that captures cross-tier dependencies between retrieval and generation, integrating parameters such as retrieval depth, document encoding cost, and decoder token rate. Complementary large-scale experiments on cloud-deployed RAG instances validate the model across multiple workloads and retriever–generator pairings. Results reveal previously unreported latency coupling effects, where minor retrieval slowdowns propagate non-linearly to generation throughput, leading to underutilized GPU compute. This work further identifies scaling laws that predict degradation thresholds and proposes lightweight scheduler adaptations that improve throughput by up to 27% under constrained latency budgets. Beyond immediate benchmarking, the framework provides a principled basis for capacity planning, SLA prediction, and energy-aware optimization of large-scale RAG systems. The study thus bridges the gap between machine-learning-centric RAG research and systems-level performance engineering, advancing both theoretical understanding and practical deployment of retrieval-generation architectures.