Measuring Stiffness in Residual Neural Networks

In this work, we define the concept of stiffness for residual neural networks (ResNets) relying on the fact that ResNets can be viewed as a discretization of an underlying neural ordinary differential equation (NODE). We then propose several metrics for the stiffness of a ResNet. We compare these measures numerically by examining their evolution over the course of training a ResNet on several test problems. We find that stiffness tends to increase as a result of training, and suggest the developed stiffness metrics can be used as training penalties, providing a novel means of regularization for ResNets.