Scaling Laws of Deep-Learning Neural Networks: Expressive Power

This chapter investigates the expressive power of neural networks through the lens of linear regions in piecewise linear neural networks (PLNNs). Quantifying these regions is vital for understanding a network’s ability to approximate complex functions; however, traditional theoretical bounds often suffer from a widening exponential gap as network capacity increases. This work aims to bridge this gap by deriving a sharp upper bound using tropical geometry. The chapter is structured into three primary methodological phases. First, it establishes the connection between PLNNs and tropical polynomials, introducing a rank-based approach for unexpanded forms that accounts for non-general hyperplane positions. Second, it proposes a precision-based approach for expanded forms, which incorporates practical constraints like weight initialization and computational precision to transform exponential growth into polynomial growth at larger widths. Finally, the chapter synthesizes these findings to derive a definitive sharp upper bound, validated through both theoretical justification and empirical analysis on real-world datasets. This contribution provides a more realistic scaling law for the representational capacity of deep learning models.