Scholarlink

<p>The vanishing gradient issue in convolutional neural networks (CNNs) is often addressed by improving activation functions, such as the S-shaped rectified linear activation unit (SReLU). However, SReLU can pose challenges in updating training parameters effectively. To mitigate this, we propose applying the Aggregation Fischer–Burmeister (AFB) function to SReLU, which smooths the secant line slope of the function from both sides. However, direct application of AFB to SReLU can intensify the vanishing gradient issue due to irregular function behavior. To address this concern, we introduce a regulated version of AFB (ReAFB) that ensures proper gradient and mean activation output conditions when applied to SReLU (ReAFBSReLU). We evaluate the performance of CNNs using ReAFBSReLU on three benchmark datasets: MNIST, CIFAR-10 (with and without data augmentation), and CIFAR-100. Specifically, we implement Network in Network (NIN) for MNIST and CIFAR-10, and LeNet for CIFAR-100 dataset. Additionally, we utilize SqueezeNet exclusively to compare the performance of CNNs using the proposed ReAFBSReLU activation function against state-of-the-art activation functions. Our results demonstrate that ReAFBSReLU outperforms other activation functions tested in this study, indicating its efficacy in enhancing training parameter updates and subsequently improving accuracy.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Effect of continuous S-shaped rectified linear function on deep convolutional neural network

  • 期刊论文

  • 发表日期：

    2025年3月15日

查看全文
  • Anahita Ghazvini,
  • Siti Norul Huda Sheikh Abdullah,
  • Masri Ayob

摘要

The vanishing gradient issue in convolutional neural networks (CNNs) is often addressed by improving activation functions, such as the S-shaped rectified linear activation unit (SReLU). However, SReLU can pose challenges in updating training parameters effectively. To mitigate this, we propose applying the Aggregation Fischer–Burmeister (AFB) function to SReLU, which smooths the secant line slope of the function from both sides. However, direct application of AFB to SReLU can intensify the vanishing gradient issue due to irregular function behavior. To address this concern, we introduce a regulated version of AFB (ReAFB) that ensures proper gradient and mean activation output conditions when applied to SReLU (ReAFBSReLU). We evaluate the performance of CNNs using ReAFBSReLU on three benchmark datasets: MNIST, CIFAR-10 (with and without data augmentation), and CIFAR-100. Specifically, we implement Network in Network (NIN) for MNIST and CIFAR-10, and LeNet for CIFAR-100 dataset. Additionally, we utilize SqueezeNet exclusively to compare the performance of CNNs using the proposed ReAFBSReLU activation function against state-of-the-art activation functions. Our results demonstrate that ReAFBSReLU outperforms other activation functions tested in this study, indicating its efficacy in enhancing training parameter updates and subsequently improving accuracy.