Scholarlink

<p>Low-Rank Adaptation (LoRA) has become a mainstream technique for fine-tuning large language models (LLMs) on downstream tasks due to its memory efficiency. However, traditional inference architectures suffer from computational redundancy and low GPU utilization when serving multi-tenant LoRA models. To address these inefficiencies, this paper presents GD2O, an SGMV-specific operator autotuning framework for multi-tenant LoRA serving. GD2O uses a graph representation to organize the construction space of the non-standard Segmented Gather Matrix-Vector Multiplication (SGMV) operator and applies a hardware-aware Markov-style stochastic search to select tiling, caching, and virtual-thread scheduling configurations. The search is used as a practical heuristic for efficient exploration, not as a formal guarantee of global optimality. The text generation experiments on Llama-2 and two GPU architectures (NVIDIA A100 and RTX 4090) demonstrate that GD2O achieves a throughput improvement of 1.29–1.52<InlineEquation ID="IEq1"><EquationSource Format="TEX">\(\times \)</EquationSource><EquationSource Format="MATHML"><math><mo>×</mo></math></EquationSource></InlineEquation> in multi-tenant LoRA serving.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Efficient multi-tenant LoRA serving via SGMV-specific operator autotuning

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月15日

查看全文
  • Chao Zhang,
  • Gang Zhou,
  • Teng Fu,
  • Jicang Lu

摘要

Low-Rank Adaptation (LoRA) has become a mainstream technique for fine-tuning large language models (LLMs) on downstream tasks due to its memory efficiency. However, traditional inference architectures suffer from computational redundancy and low GPU utilization when serving multi-tenant LoRA models. To address these inefficiencies, this paper presents GD2O, an SGMV-specific operator autotuning framework for multi-tenant LoRA serving. GD2O uses a graph representation to organize the construction space of the non-standard Segmented Gather Matrix-Vector Multiplication (SGMV) operator and applies a hardware-aware Markov-style stochastic search to select tiling, caching, and virtual-thread scheduling configurations. The search is used as a practical heuristic for efficient exploration, not as a formal guarantee of global optimality. The text generation experiments on Llama-2 and two GPU architectures (NVIDIA A100 and RTX 4090) demonstrate that GD2O achieves a throughput improvement of 1.29–1.52\(\times \)× in multi-tenant LoRA serving.