Scaling Laws of Deep-Learning Neural Networks: Taxonomy and Survey

This chapter presents a comprehensive taxonomy and survey of scaling laws in deep learning neural networks. As data-driven models grow in scale and cost, establishing theoretical estimations is crucial for guiding efficient development. Addressing the lack of a unified framework in current literature, this work introduces a “model triangular pyramid” to systematically classify scaling laws based on three fundamental dimensions: Algorithm, Data, and Computation. The chapter is structured to first analyze these factors individually, followed by an investigation of their pairwise interactions, such as sample complexity and algorithmic optimization. Finally, it explores the intersection of all three dimensions, discussing critical phenomena like the bias-variance trade-off, double descent, and scaling in Large Language Models (LLMs). This framework provides a cohesive theoretical map for navigating the complex landscape of data-driven model scaling.