In the evolving landscape of foundation models, fine-tuning pre-trained models with first-order (FO) optimizers like SGD and Adam has become standard practice. However, as these models grow in size, the substantial memory overhead from back-propagation (BP) for FO gradient computation presents a significant challenge. Addressing this issue is crucial, particularly for applications like on-device training where memory efficiency is paramount. This chapter introduces a shift towards back-propagation-free, zeroth-order (ZO) optimization as a solution to reduce memory costs during model fine-tuning. We will explore both the algorithmic foundations of ZO optimization and its applications in memory-efficient fine-tuning.