Scholarlink

<p>Convolutional neural networks (CNNs) have made tremendous progress in solving many challenging problems. Good activation functions can improve the performance of CNNs. The existing activation functions exhibit inconsistent performance gains across different training settings, models, datasets and tasks. To solve this problem, we propose a general smoothed approximation for the maximum function <InlineEquation ID="IEq1"> <InlineMediaObject> <ImageObject Color="BlackWhite" FileRef="521_2024_10935_Article_IEq1.gif" Format="GIF" Height="19" Rendition="HTML" Resolution="72" Type="Linedraw" Width="90" /> </InlineMediaObject> <EquationSource Format="TEX">\(\max (x_i, \alpha x_i)\)</EquationSource> <EquationSource Format="MATHML"><math> <mrow> <mo movablelimits="true">max</mo> <mo stretchy="false">(</mo> <msub> <mi>x</mi> <mi>i</mi> </msub> <mo>,</mo> <mi>α</mi> <msub> <mi>x</mi> <mi>i</mi> </msub> <mo stretchy="false">)</mo> </mrow> </math></EquationSource> </InlineEquation> using the linear combination of the smoothed rectified linear unit and the identity function. And we use exponential moving average to training the negative slope in this smoothed approximation. To validate the effectiveness of our approach, we also present a smoothed approximation case named leaky power function linear unit (LPFLU) to compare with the current state-of-the-art activation functions. Experimental results demonstrate that our LPFLU outperforms the existing state-of-the-art activation functions in improved robustness across different training settings, models, datasets and tasks.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Constructing a smoothed Leaky ReLU using a linear combination of the smoothed ReLU and identity function

  • 期刊论文

  • 发表日期：

    2025年1月15日

查看全文
  • Meng Zhu,
  • Weidong Min,
  • Jiahao Li,
  • Mengxue Liu,
  • Ziyang Deng,
  • Yao Zhang

摘要

Convolutional neural networks (CNNs) have made tremendous progress in solving many challenging problems. Good activation functions can improve the performance of CNNs. The existing activation functions exhibit inconsistent performance gains across different training settings, models, datasets and tasks. To solve this problem, we propose a general smoothed approximation for the maximum function \(\max (x_i, \alpha x_i)\) max ( x i , α x i ) using the linear combination of the smoothed rectified linear unit and the identity function. And we use exponential moving average to training the negative slope in this smoothed approximation. To validate the effectiveness of our approach, we also present a smoothed approximation case named leaky power function linear unit (LPFLU) to compare with the current state-of-the-art activation functions. Experimental results demonstrate that our LPFLU outperforms the existing state-of-the-art activation functions in improved robustness across different training settings, models, datasets and tasks.