DeletePop: A DLT Execution Time Predictor Based on Comprehensive Modeling

The modeling and simulation of Deep Learning Training (DLT) are challenging problems. Due to the intricate parallel patterns, existing modelings and simulations do not consider enough factors that influence the training, which brings inaccuracy for the prediction of DLT time. To address these rising challenges, we propose DeletePop, a Deep Learning Training Execution time Predictor based on comprehensive modeling at the Operator level. It systematically abstracts the process of DLT by dividing it into computation, memory access, and communication three parts. DeletePop could predict the Job Execution Time (JET) according to the operator dataset obtained from the homogeneous network. Finally, we integrate the DeletePop into a Job Scheduling Simulator (JSS) DLTSim to make support more efficient scheduling. Although the implementation of DeletePop is based on the TensorFlow framework, the theoretical model could adapt to any other frameworks that use static graphs. DeletePop achieves up to 90% accuracy for Homogeneous Networks, and we also provide the theoretical manners to add support for Heterogeneous Networks.