Scholarlink

Emerging intelligent applications often require collaborative inference from multiple deep neural networks (multi-DNNs) to support complex tasks like augmented and virtual reality. However, efficiently serving multi-DNNs is challenging due to heterogeneous model structures, parallelism strategies, and dynamic batching behaviors. Existing methods either use online task-level scheduling for batched inference or offline operator-level scheduling to optimize concurrency. These approaches, limited to a single perspective, may lead to sub-optimal performance in evolving multi-DNN serving scenarios. In this paper, we present TopServe, an efficient multi-DNN serving system that integrates dynamic batching with adaptive inter-operator parallelization strategies. During the offline phase, TopServe partitions the multi-DNN model into balanced subgraphs and generates candidate operator scheduling strategies. During the online phase, TopServe performs task-operator co-scheduling, combining effective batching with optimized operator parallelization. Our extensive evaluation shows that TopServe can significantly reduce the average latency and improve the throughput compared to state-of-the-art solutions.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

TopServe: Task-Operator Co-scheduling for Efficient Multi-DNN Inference Serving on GPUs

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Ao Chen,
  • Guangli Li,
  • Feng Yu,
  • Xueying Wang,
  • Jiacheng Zhao,
  • Huimin Cui,
  • Xiaobing Feng,
  • Jingling Xue

摘要

Emerging intelligent applications often require collaborative inference from multiple deep neural networks (multi-DNNs) to support complex tasks like augmented and virtual reality. However, efficiently serving multi-DNNs is challenging due to heterogeneous model structures, parallelism strategies, and dynamic batching behaviors. Existing methods either use online task-level scheduling for batched inference or offline operator-level scheduling to optimize concurrency. These approaches, limited to a single perspective, may lead to sub-optimal performance in evolving multi-DNN serving scenarios. In this paper, we present TopServe, an efficient multi-DNN serving system that integrates dynamic batching with adaptive inter-operator parallelization strategies. During the offline phase, TopServe partitions the multi-DNN model into balanced subgraphs and generates candidate operator scheduling strategies. During the online phase, TopServe performs task-operator co-scheduling, combining effective batching with optimized operator parallelization. Our extensive evaluation shows that TopServe can significantly reduce the average latency and improve the throughput compared to state-of-the-art solutions.