Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems

Deep learning has made significant progress in the fields of data science and natural science. Some studies have linked deep neural networks to dynamical systems, but the network structure is restricted to a residual network. It is known that residual networks can be regarded as a numerical discretization of dynamical systems. In this paper, we consider the traditional network structure and prove that vanilla feedforward networks can also be used for the numerical discretization of dynamical systems, where the width of the network is equal to the dimensions of the input and output. Our proof is based on the properties of the leaky-ReLU function and the numerical technique of the splitting method for solving differential equations. Our results could provide a new perspective for understanding the approximation properties of feedforward neural networks.