One might blame the overparameterized nature of deep learning for its lack of theoretical foundations. However, recent works have discovered that this overparameterization also contributes to the success of deep learning. Notably, there is currently no standard definition of “overparameterization”. In many cases, the definition is quite restrictive, such as requiring an infinite network width. A promising direction of future research is to relax the conditions of overparameterization.