Fuzzy Estimation of Slope Value for the LeakyReLU Applied to Convolutional Neural Networks
摘要
Within the realm of neural networks, there are various types of activation functions that can be used. One of the most used functions in convolutional neural networks is the Rectified Linear Unit (ReLU) function. This function yields good results when the network focuses on image classification. However, there is also the LeakyReLU function, which allows some negative values to pass through the activation based on a negative slope. This value is fixed and defined when creating the network architecture. Consequently, this present work is aimed at enhancing this function by altering the negative slope value based on a fuzzy system that estimates the resulting value from the input values of the previous layer. Thanks to this implementation, a mean and standard deviation of 0.989222222 and 0.0010089, respectively, were achieved using the MNIST database. Similarly, a mean and standard deviation of 0.908327778 and 0.003754048, respectively, were achieved using the Fashion-MNIST database. In both databases, better results were obtained when compared to using ReLU, where a mean and standard deviation of 0.988538889 and 0.001186355, respectively, were achieved using the MNIST database, while using Fashion-MNIST, a mean and standard deviation of 0.905855556 and 0.004285486, were achieved, respectively.