This paper proposes a deep learning platform for edge inference based on hardware/software codesign. On the hardware side, to achieve high inference throughput within the constraints of edge device, we introduce a star-style chip cluster architecture, which utilizes multiple NPUs connecting by PCIe bridge to expand deep learning computational power, and uses a CPU as the execution platform for task scheduling strategies. On the software side, hybrid parallelism scheduling strategy is designed based on data parallelism scheme on the whole and pipeline parallelism for each NPU task, so as to achieve high inference throughput. To maximize the efficiency of pipeline parallelism, we utilize the method of balancing pipeline segments. Take the most common object detection task for example, according to its processing flow and time characteristics, both pre-processing segment and post-processing segment are migrated onto CPU, and the number of threads for executing them are adjusted to balance with the intelligent processing segment. Compared with the traditional inference based on a single NPU, the system achieves the throughput acceleration ratio of 2.6X on COCO dataset, and its peak computing energy ratio is increased up to 2.5X. Last but not least, we also summarize some basic design principles based on observed experiment results, which presents useful insights for further performance improvement.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A High Throughput Deep Learning Platform for Edge Inference Based on Hardware/Software Codesign

  • Yuwei Zhang,
  • Fan Wu,
  • Chaonong Xu

摘要

This paper proposes a deep learning platform for edge inference based on hardware/software codesign. On the hardware side, to achieve high inference throughput within the constraints of edge device, we introduce a star-style chip cluster architecture, which utilizes multiple NPUs connecting by PCIe bridge to expand deep learning computational power, and uses a CPU as the execution platform for task scheduling strategies. On the software side, hybrid parallelism scheduling strategy is designed based on data parallelism scheme on the whole and pipeline parallelism for each NPU task, so as to achieve high inference throughput. To maximize the efficiency of pipeline parallelism, we utilize the method of balancing pipeline segments. Take the most common object detection task for example, according to its processing flow and time characteristics, both pre-processing segment and post-processing segment are migrated onto CPU, and the number of threads for executing them are adjusted to balance with the intelligent processing segment. Compared with the traditional inference based on a single NPU, the system achieves the throughput acceleration ratio of 2.6X on COCO dataset, and its peak computing energy ratio is increased up to 2.5X. Last but not least, we also summarize some basic design principles based on observed experiment results, which presents useful insights for further performance improvement.