This chapter focuses on optimization as the core mechanism behind deepLearningdeep learning learningDeepdeep learning. At its heart, training a neural networkNeural network involves minimizing a loss functionFunction with respect to the model’s parameters. The essential material presented in this chapter includes the foundational concept ofGradientgradient descent gradient descentDescentgradient descent and its application via the backpropagation algorithm. These form the backbone of neural networkNeural network training and are indispensable for understanding deep Learningdeep learning learningDeepdeep learning. Beyond this core material, the chapter offers several complementary sections intended to deepen understanding or provide historical and practicalPracticepractical context. These include classical optimization techniques such as line searchLine search, momentumMomentum, andSteepest descent steepest descentDescent steepest descent, as well as advanced methods like stochastic andMini-batch mini-batchBatchmini-batch gradientGradientgradient descent descentDescentgradient descent, adaptiveAdaptive learning rate Learninglearning rate algorithms (AdaGradAdaGrad, RMSPropRMSProp, and Adam)Adam, and Sharpness-Aware MinimizationSharpness-aware minimization (SAM) (SAM). While these sections are not essential for grasping the fundamentals of neural networkNeural network training, they enrich the reader’s perspective on optimization strategies used in practice. The chapter concludes with theoretical insights into the convergenceConvergence behavior of optimization algorithms, especially in the over-parameterizedOver-parameterized regime typical of deepLearningdeep learning learningDeepdeep learning. These discussions provide further context and are particularly relevant for readers interested in the theory behind the practice.