Scholarlink

<p>In recent years, a considerable number of studies have centered on enhancing the efficacy of Large Language Models (LLMs) in cross-domain tasks. Existing approaches frequently depend on large-scale data training or multi-model collaboration strategies. However, these methods encounter challenges such as imbalanced performance across domains and substantial deployment costs. To address these issues, this paper proposes a model-merging approach that integrates two domain-specific expert LLMs to train a cross-domain LLM. Specifically, we have designed a two-stage workflow, LoRA-Merge, to facilitate cross-domain training of LLMs. In the fine-tuning stage, we employ LoRA (Low-Rank Adaptation) to efficiently adapt the base LLM to the target domain. Subsequently, in the merging stage, we introduce a weight-optimization-driven merging algorithm, Slerp-Opt (Optimized Spherical Linear Interpolation), which dynamically adjusts the weight ratios of different modules within the model layers to achieve optimal integration. Slerp-Opt requires only 5% of the GPU memory and training time of LoRA fine-tuning. These improvements allow Slerp‑Opt to run comfortably within the tight memory and latency constraints of modern supercomputing hardware—delivering fast LLM merges on a single NVIDIA A40 node without additional infrastructure. Extensive experiments on diverse benchmarks demonstrate that Slerp‑Opt enhances the downstream task performance while significantly lowering resource demands. Together, these characteristics position Slerp‑Opt as an efficient, HPC‑friendly solution for fast training of large-scale LLMs.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Slerp-Opt: merging large language models via adaptive strategies

  • 期刊论文

  • 发表日期：

    2025年8月9日

查看全文
  • Haiyin Jiang,
  • Ruilin Wang,
  • Weijie Liang,
  • Qi Sun,
  • Xiang Zhang,
  • Yanan Liu

摘要

In recent years, a considerable number of studies have centered on enhancing the efficacy of Large Language Models (LLMs) in cross-domain tasks. Existing approaches frequently depend on large-scale data training or multi-model collaboration strategies. However, these methods encounter challenges such as imbalanced performance across domains and substantial deployment costs. To address these issues, this paper proposes a model-merging approach that integrates two domain-specific expert LLMs to train a cross-domain LLM. Specifically, we have designed a two-stage workflow, LoRA-Merge, to facilitate cross-domain training of LLMs. In the fine-tuning stage, we employ LoRA (Low-Rank Adaptation) to efficiently adapt the base LLM to the target domain. Subsequently, in the merging stage, we introduce a weight-optimization-driven merging algorithm, Slerp-Opt (Optimized Spherical Linear Interpolation), which dynamically adjusts the weight ratios of different modules within the model layers to achieve optimal integration. Slerp-Opt requires only 5% of the GPU memory and training time of LoRA fine-tuning. These improvements allow Slerp‑Opt to run comfortably within the tight memory and latency constraints of modern supercomputing hardware—delivering fast LLM merges on a single NVIDIA A40 node without additional infrastructure. Extensive experiments on diverse benchmarks demonstrate that Slerp‑Opt enhances the downstream task performance while significantly lowering resource demands. Together, these characteristics position Slerp‑Opt as an efficient, HPC‑friendly solution for fast training of large-scale LLMs.