Quantization Effects on a Convolutional Layer of a Deep Neural Network

Over the last few years, we have witnessed a relentless improvement in the field of computer vision and deep neural networks. In a deep neural network, convolution operation is the load bearer as it performs feature extraction and dimensionality reduction on a large scale. As the models continue to go deeper and bulkier for better efficiency and accuracy there is a rapid increment in storage requirements too. The problem arises when performing computation with efficient numerical representations for embedded devices. Transitioning from floating-point representation to fixed-point could potentially reduce computation time, storage requirements, and latency with some accuracy loss. In this paper, an analysis of the effects of quantization of the first convolutional layer on the accuracy, and memory storage requirement with varying bit-width for fixed-point integer values of network parameters has been carried out. The approach adopted is post-training quantization with a mixed-precision format to avoid model re-training and minimize accuracy loss by using root-mean-square-error (RMSE) as a performance metric. Various combination has been analyzed and compared to find the optimal precision to implement on a resource-constraint device. Based on the analysis, the suggested bit-width of I/O data for this implementation is selected as <10,5> and mid-data be <20,10> instead of <16,8> and <32,16> respectively. This combination of bit-widths has reduced memory consumption such as BRAM by 10%, DSPs by 98.6% and FFs by 40.27% with some accuracy loss.