A Redis Cache-Based Approach to High Concurrency Response in Applications of Large Language Models

The deployment of large language models (LLMs) in various domain applications has led to an urgent need for efficient response. This study proposes a Redis cache-based conversation matching approach to address the challenge of highly concurrent processing, with the aim of optimizing the power load and response speed of a large language model (LLM). The server- side maintains a cache of high-frequency requests, and responds to user requests by optimally combining the predicted content that reaches the matching threshold through the Bloom filter’s filtering algorithm and Jaccard’s similarity algorithm. This approach saves costs on API recalls and makes response times much faster. To comprehensively assess the effectiveness of this method, a series of experiments are designed that not only focus on the balance between speed and accuracy, but also clarify the best processing solution by analytically comparing different solutions side-by-side. The experiments begin with the determination of Bloom’s initial screening threshold and the Jaccard similarity threshold directly on the target text and dataset, followed by a side-by-side comparison of the accuracy and processing speed between this study’s method and other methods. The two datasets selected for the experiments include the public English dataset STS-B, which is derived from the GLUE benchmark for standard text semantic similarity evaluation, and a self-constructed Chinese text dataset covering a variety of industry domains, such as pharmaceuticals, the internet, electrical power safety, and machinery manufacturing, with a total of 3,000 samples, which is used to test the model’s ability to generalize to multii-domain text similarity understanding. The experimental results demonstrate that the method significantly improves the processing power and response speed, and effectively addresses the challenge of high concurrent response.