<p>Deep convolutional neural networks (CNNs) are a popular choice for many image classification tasks due to their good performance. In general, there is a correlation between the generalization performance of a deep learning model, the number of training samples available, and the number of free parameters in the model. If there are too many free parameters in the model relative to the amount of training data, the model overfits the training data. Reducing the number of trainable parameters in CNNs while maintaining performance levels is an active area of research. This is important, especially in resource-constrained environments where either the size of the training data is insufficient or the memory of the hardware where the model is to be deployed is limited, such as in mobile vision applications. In this paper, kernel transposition is proposed as a method for the reduction of free parameters in the model’s convolutional layers. This allows kernel reuse, in which a model learns a given kernel only once but uses it twice. The learned kernels and their transposes can either be used in sequence or in parallel to one another. These form series and parallel convolutional modules respectively. The modules are used as replacements of the traditional convolutional layers in a CNN model. The use of the modules reduces the number of free parameters and computational costs associated with convolution operations in the model without necessarily compromising its generalization performance. The proposed method is generic and can be adapted to existing state-of-the-art network architectures as demonstrated in the experiments using established architectures. The proposed method was validated on the CIFAR-10 and CIFAR-100 standard datasets using a series of experiments with five model architectures: a basic 5-convolutional layer CNN model, the ResNet-56, small MobileNetV3, EfficientNetB0 and ConvNext model architectures; all trained from scratch. Models based on standard convolutional layers (for basic CNN and ResNet-56 architectures) and depth-wise convolutional layers (for MobileNetV3, EfficientNetB0 and ConvNext architectures) were compared against those based on the proposed convolutional modules. The models based on the standard and depth-wise convolutional layers were used as baselines. The comparisons were in terms of the number of parameters, computational efficiencies (Floating Point Operations / FLOPs) and classification accuracies of the models. For a kernel size <i>k</i>, the experiments showed that, despite improvements on parameter and computational efficiencies, models based on the proposed modules for the basic CNN (<i>k</i> = 3) architecture had lower accuracy levels compared to their baseline model. However, the huge saving on the number of parameters (&gt; 88%) and FLOPs (&gt; 84%) is significant compared to the drop in accuracy (&lt; 4%). For the MobileNetV3 (<i>k</i> = 3 or <i>k</i> = 5) and EfficientNetB0 (<i>k</i> = 3 or <i>k</i> = 5), only the model based on the series module had lower accuracy than the baseline models. The models based on the parallel module were generally at par (within a margin of <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\pm\)</EquationSource> </InlineEquation> 1.5%) in terms of accuracy with their respective baselines. In relation to the ConvNext (<i>k</i> = 7) and the ResNet-56 (<i>k</i> = 3 or <i>k</i> = 11, or <i>k</i> = 19) architectures, other than the improved parameter and computational efficiencies, the classification accuracies of all the models based on the proposed modules were either at par or better than those of their baseline models. In all the above architectures, the use of the proposed modules reduced the number of convolutional parameters in the networks by at least 78% with fewer FLOPs relative to the baseline models</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Parameter reduction in convolutional neural networks with kernel transposition

  • Daniel Arani Osuto,
  • Heywood Absaloms Ouma,
  • Wilfred Njoroge Mwema

摘要

Deep convolutional neural networks (CNNs) are a popular choice for many image classification tasks due to their good performance. In general, there is a correlation between the generalization performance of a deep learning model, the number of training samples available, and the number of free parameters in the model. If there are too many free parameters in the model relative to the amount of training data, the model overfits the training data. Reducing the number of trainable parameters in CNNs while maintaining performance levels is an active area of research. This is important, especially in resource-constrained environments where either the size of the training data is insufficient or the memory of the hardware where the model is to be deployed is limited, such as in mobile vision applications. In this paper, kernel transposition is proposed as a method for the reduction of free parameters in the model’s convolutional layers. This allows kernel reuse, in which a model learns a given kernel only once but uses it twice. The learned kernels and their transposes can either be used in sequence or in parallel to one another. These form series and parallel convolutional modules respectively. The modules are used as replacements of the traditional convolutional layers in a CNN model. The use of the modules reduces the number of free parameters and computational costs associated with convolution operations in the model without necessarily compromising its generalization performance. The proposed method is generic and can be adapted to existing state-of-the-art network architectures as demonstrated in the experiments using established architectures. The proposed method was validated on the CIFAR-10 and CIFAR-100 standard datasets using a series of experiments with five model architectures: a basic 5-convolutional layer CNN model, the ResNet-56, small MobileNetV3, EfficientNetB0 and ConvNext model architectures; all trained from scratch. Models based on standard convolutional layers (for basic CNN and ResNet-56 architectures) and depth-wise convolutional layers (for MobileNetV3, EfficientNetB0 and ConvNext architectures) were compared against those based on the proposed convolutional modules. The models based on the standard and depth-wise convolutional layers were used as baselines. The comparisons were in terms of the number of parameters, computational efficiencies (Floating Point Operations / FLOPs) and classification accuracies of the models. For a kernel size k, the experiments showed that, despite improvements on parameter and computational efficiencies, models based on the proposed modules for the basic CNN (k = 3) architecture had lower accuracy levels compared to their baseline model. However, the huge saving on the number of parameters (> 88%) and FLOPs (> 84%) is significant compared to the drop in accuracy (< 4%). For the MobileNetV3 (k = 3 or k = 5) and EfficientNetB0 (k = 3 or k = 5), only the model based on the series module had lower accuracy than the baseline models. The models based on the parallel module were generally at par (within a margin of \(\pm\) 1.5%) in terms of accuracy with their respective baselines. In relation to the ConvNext (k = 7) and the ResNet-56 (k = 3 or k = 11, or k = 19) architectures, other than the improved parameter and computational efficiencies, the classification accuracies of all the models based on the proposed modules were either at par or better than those of their baseline models. In all the above architectures, the use of the proposed modules reduced the number of convolutional parameters in the networks by at least 78% with fewer FLOPs relative to the baseline models