Learning N:M Structured Sparse Neural Networks from Scratch: A Comparative Study

Sparsity in deep neural networks (DNNs) has attracted significant attention in recent years due to its ability to compress DNN models and speed up their execution in resource-constrained environments. Among different approaches, N:M structured sparsity has taken the lead thanks to its efficient support in modern hardware accelerators. While numerous training methods for N:M sparse DNNs have been proposed, there is a lack of comparisons among these methods, which prevents their fair and consistent evaluations. To address such a gap, this paper presents a comparative study of some state-of-the-art methods for training N:M structured sparse neural networks. We first start by proposing a taxonomy of N:M structured pruning configuration settings to systematically categorize and organize these methods. Building upon this taxonomy, we conduct several experiments to, qualitatively and quantitatively, compare the prominent works in the field under a uniform set of conditions. Our findings highlight opportunities for improving sparse neural network performance through approaches involving layer-wise N:M sparse patterns, pretrained weight initialization, suitable pruning ratios, and the selection of effective sparse training strategies.