Efficient Deep Learning Inference Engine on SoC Platform

Acceleration of Deep Neural Networks (DNN) is essential for various mission-critical applications. Though deep learning provides good accuracy, it involves various computationally intensive mathematical operations like convolution, multiplication, etc. These complex computations not only consume large hardware resources but also increase the latency. Various Deep Network models like LeNet, GoogleNet, VGGNet, AlexNet, etc. have the same operational layers like convolution, pooling, and activation. There are different platforms to implement hardware accelerators like Application-Specific Integrated Circuits (ASIC), Field Programmable Gate Array (FPGA), Graphics Processing Unit (GPU), or System-on-Chip (SoC). Due to inherent parallelism, low non-recurring engineering cost, and on-field programmability, FPGAs provide good solutions for the implementation of hardware accelerators compared to other platforms. In this work, we have developed and integrated an inference engine consisting of a task tree extractor module, a scheduler, and a deep learning accelerator using a hardware-software co-design paradigm on SoC for acceleration of the deep neural networks. Using the proposed system, the task tree for a Lenet-5 and VGG-16 model has been generated in 0.75 and 20.5 s with an accuracy of 97.6 and 90.75 respectively on ZYNQ Ultrascale+ MPSoC ZCU102.