MJOS: A Multi-stage Joint Optimization Strategy for Convolutional Neural Network Compression

Employing a blend of multiple, mutually reinforcing compression techniques proves more effective in lightening convolutional neural network (CNN) models than relying solely on one technique. However, the combination of disparate techniques can introduce inherent conflicts without meticulous design and integration strategies, leading to suboptimal outcomes. In this study, we introduce MJOS, a Multi-stage Joint Optimization Strategy tailored for CNN compression. MJOS comprises two compression stages, succeeded by an accuracy recovery stage. During the compression stages, it synergizes filter pruning with two unique tensor-train low-rank approximation techniques. These techniques collectively reduce parameters across diverse compressible dimensions, encompassing convolutional layer channels, fully connected layer channels, and kernel sizes. Subsequently, in the accuracy recovery stage, it designates the compressed model as the student network and leverages knowledge distillation to restore accuracy. Extensive experiments across various CNN architectures and benchmark datasets underscore the superiority of MJOS compared to leading-edge techniques, reinforcing the indispensability of each constituent technique.