On the Training Efficiency of Shallow Architectures for Physics Informed Neural Networks

Physics-informed Neural Networks (PINNs), a class of neural models that are trained by minimizing a combination of the residual of the governing partial differential equation and the initial and boundary data, have gained immense popularity in the natural and engineering sciences. Despite their observed empirical success, an analysis of the training efficiency of residual-driven PINNs at different architecture depths is poorly documented. Usually, neural models used for machine learning tasks such as computer vision and natural language processing have deep architectures, that is, a larger number of hidden layers. In PINNs, we show that for a given trainable parameter count (model size), a shallow network (less layers) converges faster than a deep network (more layers) for the same error characteristics. To illustrate this, we examine the one-dimensional Poisson’s equation and evaluate the gradient for residual and boundary loss terms. We show that the characteristics of the gradient of the loss function are such that for residual loss, shallow architectures converge faster. Empirically, we show the implications of our theory through various experiments.