Scholarlink

<p>This work provides a comprehensive theoretical and empirical analysis of <i>SwitchPath</i>, a stochastic activation function that improves learning dynamics by probabilistically toggling between a neuron standard activation and its negation. We develop theoretical foundations and demonstrate its impact in multiple scenarios. By maintaining gradient flow and injecting controlled stochasticity, the method improves generalization, uncertainty estimation, and training efficiency. Experiments in classification show consistent gains over ReLU and Leaky ReLU across CNNs and Vision Transformers, with reduced overfitting and better test accuracy. In generative modeling, a novel two-phase training scheme significantly mitigates mode collapse and accelerates convergence. Our theoretical analysis reveals that SwitchPath introduces a form of multiplicative noise that acts as a structural regularizer. Additional empirical investigations show improved information propagation and reduced model complexity. These results establish this activation mechanism as a simple yet effective way to enhance exploration, regularization, and reliability in modern neural networks.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Exploration and generalization in deep learning with SwitchPath activations

  • 期刊论文

  • 发表日期：

    2025年8月5日

查看全文
  • Antonio Di Cecco,
  • Andrea Papini,
  • Carlo Metta,
  • Marco Fantozzi,
  • Silvia Giulia Galfrè,
  • Francesco Morandin,
  • Maurizio Parton

摘要

This work provides a comprehensive theoretical and empirical analysis of SwitchPath, a stochastic activation function that improves learning dynamics by probabilistically toggling between a neuron standard activation and its negation. We develop theoretical foundations and demonstrate its impact in multiple scenarios. By maintaining gradient flow and injecting controlled stochasticity, the method improves generalization, uncertainty estimation, and training efficiency. Experiments in classification show consistent gains over ReLU and Leaky ReLU across CNNs and Vision Transformers, with reduced overfitting and better test accuracy. In generative modeling, a novel two-phase training scheme significantly mitigates mode collapse and accelerates convergence. Our theoretical analysis reveals that SwitchPath introduces a form of multiplicative noise that acts as a structural regularizer. Additional empirical investigations show improved information propagation and reduced model complexity. These results establish this activation mechanism as a simple yet effective way to enhance exploration, regularization, and reliability in modern neural networks.