Networks-on-Chip to Enable Large-Scale Multi-core ML Acceleration

Emerging deep neural network (DNN) applications require high performance and benefits from heterogeneous multi-core hardware acceleration as evidenced by the implementations of TinyVers (Chap. 6 ) and DIANA (Chap. 7 ). Future systems will draw on this heterogeneous trend and are envisioned to grow to many more central processing unit (CPU) and accelerator cores (Verhelst et al., IEEE Solid State Circuits Magazine 14(4):18–27, 2022). When moving to large-scale multi-core accelerations, data communication can quickly become a bottleneck preventing the full utilization of the multiple cores. Traditionally in multi-core general-purpose CPU-based systems, networks-on-chip (NoC) have been used to move data efficiently. However, networks-on-chip designed for CPU traffic suffer from protocol translation overhead between the standard channel-oriented protocols at the endpoints (e.g., AXI4 or AXI5) and the serialized and packetized protocol typical for the networks-on-chip (NoC)s. Specifically, the burst-based deterministic traffic in deep neural networks (DNN) acceleration platforms leads to large area overhead and low bandwidth performance. As a result, many DNN platforms fall back to ad hoc networks, trading off NoC flexibility to gain performance. To better address the specific needs of emerging multi-core DNN computing platforms, this chapter proposes PATRONoC, an open-source fully AXI-compliant NoC fabric. The design takes advantage of the AXI protocol to build a parametrizable, multi-channel, and burst-oriented NoC, providing an easy plug-and-play fabric for AXI endpoints, leading to a fully protocol-homogeneous network. Evaluation of PATRONoC in a 2D mesh topology shows that the area efficiency is 34% higher than a comparable state-of-the-art traditional NoC, both running at 1 GHz clock frequency in a 22 nm FDX technology node. In terms of throughput, PATRONoC outperforms baseline NoCs in the Noxim simulator by 2–8 \(\times \) for direct memory access (DMA) bursts of uniform random traffic. The NoC design also provides high aggregated throughput from a few GiB/s to 350 GiB/s on both synthetic and DNN workload traffic patterns.