Scholarlink

<p>In recent years, large language model (LLM) technologies have achieved rapid advancements. As the scale of model parameters continues to grow, training LLMs imposes higher demands and challenges on data storage capacity and access efficiency. By analyzing training workloads and LLM I/O characteristics, our paper reveals the main bottlenecks of traditional storage systems in LLM training, including prolonged checkpointing interruptions, low failure recovery efficiency, and inadequate data loading performance. In response to these challenges, we introduce LMStor, a storage optimization framework for LLM training. LMStor proposes a fully asynchronous hierarchical checkpointing technology, which leverages asynchronous GPU-to-host memory copying and dual-replica storage at the cache layer. It also incorporates a multi-level failure recovery strategy based on cache consistency, allowing optimal recovery solutions tailored to the failure level. Additionally, LMStor implements a dataset loading acceleration technology, using optimized prefetch strategies and multi-level cache mechanisms. Experimental results demonstrate that LMStor reduces checkpoint interruptions from minutes to milliseconds, improves failure recovery efficiency by 3 to 24 times compared to traditional methods, and increases dataset loading efficiency by 46.63% compared to Alluxio. In high-frequency checkpointing scenarios of LLaMA2-70B, compared to Megatron, synchronous solution with Alluxio, and asynchronous solution combining Alluxio and DLRover, LMStor reduces the total training time by 39.9%, 20.7% and 5.1%, respectively. Hence, LMStor significantly improves storage access performance and computing resource utilization in LLM training.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Research on storage optimization for efficient training of large language models

  • 期刊论文

  • 发表日期：

    2025年8月1日

查看全文
  • Biyun Shang,
  • Feng Zhang,
  • Mo Xu,
  • Junning Xu,
  • Xinyuan Sun,
  • Zhenjiang Dong

摘要

In recent years, large language model (LLM) technologies have achieved rapid advancements. As the scale of model parameters continues to grow, training LLMs imposes higher demands and challenges on data storage capacity and access efficiency. By analyzing training workloads and LLM I/O characteristics, our paper reveals the main bottlenecks of traditional storage systems in LLM training, including prolonged checkpointing interruptions, low failure recovery efficiency, and inadequate data loading performance. In response to these challenges, we introduce LMStor, a storage optimization framework for LLM training. LMStor proposes a fully asynchronous hierarchical checkpointing technology, which leverages asynchronous GPU-to-host memory copying and dual-replica storage at the cache layer. It also incorporates a multi-level failure recovery strategy based on cache consistency, allowing optimal recovery solutions tailored to the failure level. Additionally, LMStor implements a dataset loading acceleration technology, using optimized prefetch strategies and multi-level cache mechanisms. Experimental results demonstrate that LMStor reduces checkpoint interruptions from minutes to milliseconds, improves failure recovery efficiency by 3 to 24 times compared to traditional methods, and increases dataset loading efficiency by 46.63% compared to Alluxio. In high-frequency checkpointing scenarios of LLaMA2-70B, compared to Megatron, synchronous solution with Alluxio, and asynchronous solution combining Alluxio and DLRover, LMStor reduces the total training time by 39.9%, 20.7% and 5.1%, respectively. Hence, LMStor significantly improves storage access performance and computing resource utilization in LLM training.