Maximum Entropy Learning with Neural Networks

Conventionally, the back-propagation (BP), maximum likelihood (ML) and Bayesian approaches have been applied to train Artificial Neural Networks (ANN). This study presents a Generalized Maximum Entropy (GME) learning algorithm for ANN, designed specifically to handle limited training data and unknown error distribution. Maximizing only the entropy of parameters in the ANN allows more effective generalization capability, less bias towards data distributions, and robustness to over-fitting compared to the conventional algorithm learning. In the implementations, GME is compared with the conventional algorithms in terms of their forecasting performances in both simulation and real data studies. The findings demonstrate that GME outperforms other competing estimators when training data is limited and the distribution of the error is unknown.