Accurate Latency Prediction of Deep Learning Model Inference Under Dynamic Runtime Resource

Accurate prediction of inference time can effectively accelerate model design and deployment in neural architecture search (NAS) algorithms, and provide hints for process scheduling in intelligent systems. Most of current latency prediction methods assume that all hardware resources are dominated by a single inference task. Due to the neglect of inter-process resource contention, these methods will fail in multi-tasking parallel scenarios. As an attempt to fill this gap, a hardware-aware prediction method based on double feature embedding (HADE) for deep learning model inference time is proposed. In HADE, operator-wise switchable Multilayer Perceptron and GNN-based graph embedding are used to encode node features, which are generated by unified nodes vectorization. Then, relying on the Resource Aware Latency (RAL) formula based on processor modeling techniques, the model inference time under dynamic runtime resource can be predicted. The experiments show that the accuracy of standard inference time prediction can reach 96.4%, and of resource dependent inference time can reach 79.7%.