Why Skew-Normal Distributions and How They Are Related to ReLU Activation Function in Deep Learning
摘要
Normal distributions are ubiquitous, but many actual distributions are different from normal—for example, they are skewed. To describe such distributions, it is desirable to have a few-parametric family that extends the family of normal distributions. Several such families have been proposed. Empirically, the most effective among them is the family of so-called skew-normal distributions first proposed by A. Azzalini. In particular, this family is effective in econometrics. In this paper, we provide a theoretical explanation for this empirical success. This explanation is similar to an explanation of what ReLU activations functions are most effective in deep learning.