The remarkable generalization capabilities of large language models (LLMs), along with their expert-level performance across various domains, have positioned them as a key driver of the next technological revolution, leading to their widespread deployment. However, the presence of the Key-Value(KV) cache introduces significant challenges in efficient memory management, which is crucial for deploying LLMs effectively.To address this, dynamic memory management solutions such as PagedAttention have been proposed to allocate GPU memory on demand. However, to mitigate the risk of out-of-memory (OOM) errors, some requests must be suspended and later reactivated when sufficient memory becomes available. This reactivation process, however, incurs substantial recovery latency, leading to inference delays. To reduce recovery latency, we propose Slime, an adaptive hybrid recovery strategy that dynamically partitions recovery tasks into Recomputation and Load operations based on predictor-guided decisions. By leveraging the computation-communication overlap of GPUs, Slime significantly minimizes recovery latency. Experimental results show that Slime reduces recovery latency by approximately 40% compared to the best-performing single recovery strategy, while consistently selecting the optimal recovery approach across all scenarios.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Slime: An Intelligent Hybrid Recovery Strategy that Minimizes Recovery Overhead

  • Yuwen Wang,
  • Wuhui Chen

摘要

The remarkable generalization capabilities of large language models (LLMs), along with their expert-level performance across various domains, have positioned them as a key driver of the next technological revolution, leading to their widespread deployment. However, the presence of the Key-Value(KV) cache introduces significant challenges in efficient memory management, which is crucial for deploying LLMs effectively.To address this, dynamic memory management solutions such as PagedAttention have been proposed to allocate GPU memory on demand. However, to mitigate the risk of out-of-memory (OOM) errors, some requests must be suspended and later reactivated when sufficient memory becomes available. This reactivation process, however, incurs substantial recovery latency, leading to inference delays. To reduce recovery latency, we propose Slime, an adaptive hybrid recovery strategy that dynamically partitions recovery tasks into Recomputation and Load operations based on predictor-guided decisions. By leveraging the computation-communication overlap of GPUs, Slime significantly minimizes recovery latency. Experimental results show that Slime reduces recovery latency by approximately 40% compared to the best-performing single recovery strategy, while consistently selecting the optimal recovery approach across all scenarios.