As the scale of deep learning models and datasets expands rapidly, the use of distributed training, especially model parallel training, is necessary to enhance the training performance of large-scale neural network models. However, due to limitations in computational and storage resources as well as model size, how to automate the parallel decomposition has become an important challenge. Current research achievements have made some progress in simplifying the design of parallel strategies, but there are certain limitations, including a tendency to fall into local optima and low search efficiency. To address these issues, this paper proposes an Improved Dual Population Genetic Algorithm (IDPGA) to optimize the application of TGA in automatic model parallelization. IDPGA enhances the global search capability of the algorithm, improves search diversity, and reduces the problem of local optima by introducing strategies such as retaining elite individuals, dynamic migration operators, and optimized sorting algorithms. We have also constructed a fine-grained multi-dimensional cost evaluation model, which can more comprehensively assess the merits of parallel strategies and significantly improve strategy execution performance and search efficiency through a dual-population iterative search mechanism. Experimental results indicate that the improved dual population genetic algorithm method (IDPGA) can automate the implementation of parallel decomposition strategies, with a significant increase in training efficiency.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Parallel Decomposition Method for Deep Learning Models Based on Improved Dual Population Genetic Algorithm

  • Zi Han,
  • Zhengyang Xu,
  • Tianyi Wang,
  • Kaidi Wang

摘要

As the scale of deep learning models and datasets expands rapidly, the use of distributed training, especially model parallel training, is necessary to enhance the training performance of large-scale neural network models. However, due to limitations in computational and storage resources as well as model size, how to automate the parallel decomposition has become an important challenge. Current research achievements have made some progress in simplifying the design of parallel strategies, but there are certain limitations, including a tendency to fall into local optima and low search efficiency. To address these issues, this paper proposes an Improved Dual Population Genetic Algorithm (IDPGA) to optimize the application of TGA in automatic model parallelization. IDPGA enhances the global search capability of the algorithm, improves search diversity, and reduces the problem of local optima by introducing strategies such as retaining elite individuals, dynamic migration operators, and optimized sorting algorithms. We have also constructed a fine-grained multi-dimensional cost evaluation model, which can more comprehensively assess the merits of parallel strategies and significantly improve strategy execution performance and search efficiency through a dual-population iterative search mechanism. Experimental results indicate that the improved dual population genetic algorithm method (IDPGA) can automate the implementation of parallel decomposition strategies, with a significant increase in training efficiency.