How to Train Accurate BNNs for Embedded Systems?

A key enabler of deploying convolutional neural networks on resource-constrained embedded systems is the binary neural network (BNN). BNNs save on memory and simplify computation by binarizing both features and weights. Unfortunately, binarization is accompanied by a severe decrease in accuracy. To reduce the accuracy gap between binary and full-precision networks, many repair methods have been proposed in the recent past, which we have classified and put into a single overview. The repair methods are divided into two main branches, training techniques and network topology changes, which are further split into smaller categories. We observe progress in reducing the accuracy gap, but BNN papers are not aligned on what repair methods should be used. Therefore, we conduct an empirical review that evaluates the benefits of repair methods in isolation on two benchmarks: CIFAR10 on ResNet-20 and CIFAR100 on ResNet-18. We found three repair categories most beneficial: feature binarizer, feature normalization, and double residual. Based on this review, we discuss future directions and research opportunities. We sketch the benefit and costs associated with BNNs on embedded systems because it remains to be seen whether BNNs will be able to close the accuracy gap while staying highly energy-efficient on resource-constrained embedded systems.