In cloud-native AI inference services, model loading has become a dominant contributor to end-to-end latency, and its efficiency directly influences system responsiveness, service quality, and user experience. Despite the elasticity of cloud resources, loading performance remains highly sensitive to cache pool allocation, container bandwidth, and cloud disk configurations. Inefficient resource provisioning often leads to prolonged startup time and excessive operation cost, making it difficult to consistently satisfy stringent Service Level Objectives (SLOs) under dynamic and large-scale deployments. To address these challenges, we present CCOptimizer, a cloud-native cache optimizer framework tailored for AI inference services. We introduce the core features and underlying techniques of CCOptimizer, and we demonstrate its effectiveness on a real-world multi-node Kubernetes cluster with representative foundation models.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

CCOptimizer: Resource Configuration Optimizer for Model Cache Pool in Cloud

  • Qiming Chen,
  • Rong Gu,
  • Guoding Ji,
  • Chengying Huan,
  • Yan Ding,
  • Chaozhong Yan,
  • Gang Huang

摘要

In cloud-native AI inference services, model loading has become a dominant contributor to end-to-end latency, and its efficiency directly influences system responsiveness, service quality, and user experience. Despite the elasticity of cloud resources, loading performance remains highly sensitive to cache pool allocation, container bandwidth, and cloud disk configurations. Inefficient resource provisioning often leads to prolonged startup time and excessive operation cost, making it difficult to consistently satisfy stringent Service Level Objectives (SLOs) under dynamic and large-scale deployments. To address these challenges, we present CCOptimizer, a cloud-native cache optimizer framework tailored for AI inference services. We introduce the core features and underlying techniques of CCOptimizer, and we demonstrate its effectiveness on a real-world multi-node Kubernetes cluster with representative foundation models.