Optimization of Echo State Neural Networks to Solve Classification Problems

Echo State Neural Networks (ESNN) are a kind of recurrent neural networks, which are computationally very cheap to train: input and reservoir weights are initialized randomly, the output weights are calculated by linear regression. The cost of linear regression is equivalent to invert a matrix. In this work, the five parameters of an ESNN, input noise, leaking rate, reservoir size, the seed of the random generator for the reservoir weights initialization, and the seed for a permutation of the reservoir weights, are optimized by the heuristics genetic algorithm and pattern search to solve classification tasks. We show that with these five parameters one instance of an ESNN is completely characterized. With these five parameters it is not possible to perform an exhaustive search of their values. We show with several public datasets the results of the optimization and the comparison with the manual optimization of only the reservoir size.