SW-LeNet: Implementation and Optimization of LeNet-1 Algorithm on Sunway Bluelight II Supercomputer

Nowadays, convolutional neural networks are representative of deep learning algorithms. With the development of convolutional neural networks, their network structures become more complex, and the number of parameters for training becomes larger and larger. The parallelization of convolutional neural network algorithms on multicore or many-core processors is essential for training convolutional neural networks. In this paper, we propose a parallel algorithm of LeNet-1 based on the Sunway Bluelight II supercomputer, named SW-LeNet. Moreover, we propose a two-level parallelization scheme, including thread-level optimization and process-level optimization. In thread-level optimization, the following optimization methods are used, including CPEs parallelism, hybrid scheme and DMA optimization, register optimization, and SIMD data parallelism. Data parallelism optimization and parameter packing optimization are used in process-level optimization. Compared with the original LeNet, SW-LeNet can achieve 4.94x speedups in a single core group. Moreover, SW-LeNet can be scaled up to 2,048 processes, with 133,120 cores, and achieves 84.93% parallel efficiency.