Software Framework for Embedded Neural Networks

This chapter introduces common embedded neural network software frameworks in detail. Some of them are universal and can be adapted to various AI acceleration chips, such as TensorFlow Lite, Apache TVM, etc., while others are developed for specific AI acceleration chips, such as TensorRT, OpenVINO, XILINX Vitis, uTensor, Qualcomm AI stack, etc. For each framework, its main functions, module composition, workflow, etc. are introduced. Finally, each framework is compared in terms of supported AI chip types and supported neural network training frameworks to facilitate developers to choose according to their needs. As AI becomes a major driver of edge technologies, the combination of hardware accelerators and software platforms becomes increasingly important for running inference models. Some hardware accelerators have launched supporting software development frameworks, such as NVIDIA’s TensorRT. There are also some manufacturers that have launched general embedded neural network software frameworks, such as TensorFlow Lite. They are introduced separately below.