Millisecond-Level Interference-Aware Scheduling for Multi-Inference Co-Location on Ascend NPUs
摘要
The growing popularity of AI inference services has created a substantial demand for AI accelerators. However, the strict tail-latency requirements of inference workloads often conflict with the throughput optimization objectives of these accelerators. Many AI accelerators (e.g., Ascend NPUs, Kunlun chips) support co-located deployment of inference tasks via temporal sharing to improve throughput, where interference between tasks can be abstracted as kernel queuing delays. Motivated by this observation, we design nShare, a system that detects interference at the kernel level on hardware supporting temporal sharing and performs millisecond-scale scheduling control. nShare models interference on temporal shared devices as SLO (Service Level Object) slack and incorporates a dynamic batching mechanism to improve utilization without violating latency constraints. Compared to baseline systems, nShare improves throughput by 39.48%–51.53% while meeting the 99th-percentile SLO.