DeepLat: Achieving Minimum Worst Case Latency for DNN Inference with Batch-Aware Dispatching

Deep neural network (DNN) has achieved the state-of-the-art results in multiple fields, and has been widely used to build latency sensitive applications for its high performance. When dispatching requests among GPU machines for DNN execution, the inference system hosted in the cloud needs to guarantee that the maximum latency of all requests, denoted as the worst case latency, is within the latency objectives of the clients. In this paper, we design and implement a request dispatch system, called DeepLat, which distributes client requests among the GPU machines efficiently to minimize the worst case latency of the DNN-based application. DeepLat uses batch-aware dispatch policy to minimize the batch collecting time, proposes duration-based algorithm to reduce the average latency and supports partial-batch dispatching to minimize the waiting time for bottleneck machines. Evaluation shows that compared to existing request dispatch systems, DeepLat can reduce the worst case latency by \(37.7\%\) on average without using extra computing resources. Besides, DeepLat achieves the theoretical lower bound for the worst case latency for over \(48\%\) workload. With the capability to minimize the worst case latency, DeepLat reduces the total cost of DNN serving system by \(43.2\%\) on average.