Scholarlink

In real-world applications of Deep Learning (DL), where multi-variate data is most prominently processed, the choice of Activation Function (AF) plays a critical role for reliable generalization and good convergence. Traditional AFs such as Rectified Linear Unit (ReLU) and its advanced variants such as Leaky-ReLU (L-ReLU), and Parametric-ReLU (P-ReLU) are most commonly used due to their simplicity and effectiveness. However, these functions are inherently unbounded and non-smooth, which can lead to unstable gradients with unoptimized convergence and therefore poor generalization, particularly when processing chaotic, large magnitude floating point values. Additionally, extreme values combined with steep slopes can trigger chain reaction of overcompensation, further destabilizing the learning process. To address these issues, while retaining the simplicity of the widely as-default preferred ReLU AF and its derivatives, we propose Tanned-ReLU (Td-ReLU), a novel AF for artificial neural networks that combines the hyperbolic tangent (tanh) and ReLU for smooth and bounded outputs for volatile inputs. The proposed AF is evaluated on a diverse set of real-world multivariate datasets with high relevance to business critical decision making domain, including 5G Quality of Service, wind power forecasting, and IoT based water quality monitoring. Using DL models such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models, Td-ReLU consistently outperformed state-of-the-art ReLU and its counterparts (L-ReLU and P-ReLU); with substantial improvements in Relative Root Mean Squared Error (RRMSE), showing reductions of up to 82.44% compared to P-ReLU, up to 5.18% compared to L-ReLU, and up to 1.07% compared to ReLU across said datasets. These results underline Td-ReLU’s consistency and generalization ability, confirming it as a strong alternative to traditional activation functions for applications involving complex temporal and multivariate data, qualities that are highly valuable in enterprise scale DL systems for deployment and decision making.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Tanned-ReLU: A Bounded and Smooth Activation Function for Reliable Deep Learning on Chaotic Multi-variate Data

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Saifullah Khan,
  • Keijo Haataja,
  • Pekka Toivanen

摘要

In real-world applications of Deep Learning (DL), where multi-variate data is most prominently processed, the choice of Activation Function (AF) plays a critical role for reliable generalization and good convergence. Traditional AFs such as Rectified Linear Unit (ReLU) and its advanced variants such as Leaky-ReLU (L-ReLU), and Parametric-ReLU (P-ReLU) are most commonly used due to their simplicity and effectiveness. However, these functions are inherently unbounded and non-smooth, which can lead to unstable gradients with unoptimized convergence and therefore poor generalization, particularly when processing chaotic, large magnitude floating point values. Additionally, extreme values combined with steep slopes can trigger chain reaction of overcompensation, further destabilizing the learning process. To address these issues, while retaining the simplicity of the widely as-default preferred ReLU AF and its derivatives, we propose Tanned-ReLU (Td-ReLU), a novel AF for artificial neural networks that combines the hyperbolic tangent (tanh) and ReLU for smooth and bounded outputs for volatile inputs. The proposed AF is evaluated on a diverse set of real-world multivariate datasets with high relevance to business critical decision making domain, including 5G Quality of Service, wind power forecasting, and IoT based water quality monitoring. Using DL models such as Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) models, Td-ReLU consistently outperformed state-of-the-art ReLU and its counterparts (L-ReLU and P-ReLU); with substantial improvements in Relative Root Mean Squared Error (RRMSE), showing reductions of up to 82.44% compared to P-ReLU, up to 5.18% compared to L-ReLU, and up to 1.07% compared to ReLU across said datasets. These results underline Td-ReLU’s consistency and generalization ability, confirming it as a strong alternative to traditional activation functions for applications involving complex temporal and multivariate data, qualities that are highly valuable in enterprise scale DL systems for deployment and decision making.