Pioneer: A Method for Enhancing GPU Utilization in Workloads Through Potential-Aware Prioritization
摘要
With the rapid advancement of Large Language Models (LLMs), conventional multimodal training frameworks often fail to fully utilize computational resources. Current research typically enhances the training process through improved memory management, task partitioning, and dynamic scheduling. However, this paper emphasizes that scheduling priorities also play a crucial role in resource utilization. We introduce a multimodal, multitask training system based on potential-aware prioritization, named Pioneer. This system integrates heterogeneous load characteristics with dependencies in the computational graph, mapping scheduling priorities by evaluating the potential value of each operation. Operations with higher potential are prioritized, which in turn releases more downstream tasks and maximizes GPU utilization efficiency. Specifically, to address the heterogeneity of computational loads, operations with similar loads are grouped into OpGroup. We employ a hierarchical multi-metric modeling approach to segmentally fit different metric coefficients, node entropy, and critical path lengths, enabling precise evaluation of OpGroup ’s computational resources and potential values. Leveraging these outputs, combined with a Stage-greedy scheduling strategy, we optimize resource allocation during the training process using Stage as the scheduling unit. Our system effectively avoids the decline in GPU utilization caused by random resource allocation in multiple scheduling rounds. Comparative experiments show that Pioneer significantly enhances resource utilization while handling complex multitasks, thus optimizing training performance.