With the explosive growth of the parameter scale of large language models (LLMs), the computing resource and communication overhead required for their training have increased dramatically. While the existing distributed training framework has significant deficiencies in resource scheduling, parallel strategies coordination and communication process modeling. This paper proposes a computing communication modeling method and simulation system for distributed training of LLMs. Through systematic analysis and detailed quantification, we build accurate computing load and communication process model, providing solid data support for LLMs training optimization; at the same time, a simulation environment is designed and implemented, which can simulate the distributed training environment of 64 GPUs and accurately reproduce the communication mechanism and process of LLMs training traffic. In the experiment, the computing and communication process of the 0.01B LLMs was modeled and quantified, and the performance of the packet sending end in the simulation environment was tested. The results show that the packet sending end can achieve a stable packet sending rate and precise time control. This paper provides theoretical support and practical basis for the communication optimization of LLMs distributed training cluster.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Computational Communication Modeling Method and Simulation System for Large Language Model Distributed Training

  • Yuanhao He,
  • Yimin Liu,
  • Hanguang Luo,
  • Jun Zhu,
  • Tao Zou,
  • Juncheng Ge,
  • Sheng Li

摘要

With the explosive growth of the parameter scale of large language models (LLMs), the computing resource and communication overhead required for their training have increased dramatically. While the existing distributed training framework has significant deficiencies in resource scheduling, parallel strategies coordination and communication process modeling. This paper proposes a computing communication modeling method and simulation system for distributed training of LLMs. Through systematic analysis and detailed quantification, we build accurate computing load and communication process model, providing solid data support for LLMs training optimization; at the same time, a simulation environment is designed and implemented, which can simulate the distributed training environment of 64 GPUs and accurately reproduce the communication mechanism and process of LLMs training traffic. In the experiment, the computing and communication process of the 0.01B LLMs was modeled and quantified, and the performance of the packet sending end in the simulation environment was tested. The results show that the packet sending end can achieve a stable packet sending rate and precise time control. This paper provides theoretical support and practical basis for the communication optimization of LLMs distributed training cluster.