Zeroth-order (ZO) optimization has become a popular technique for solving machine learning problems, especially when first-order (FO) information is difficult or impossible to obtain. However, there remains a lack of extensive surveys covering recent advances and thorough analysis of ZO optimization for machine learning. In this paper, we present a comprehensive survey on ZO optimization, aiming to summarize the cutting-edge research and broaden the horizons for different domains. Firstly, we introduce the definition of ZO optimization by characterizing its inherent properties with respect to machine learning problems. In particular, we dig out some critical ingredients from the properties including no gradient dependence, dimension sensitivity, noise robustness, and implicit smoothing. Second, we summarize emerging strategies by combining the properties of ZO optimization and specific machine learning problems like adversarial examples, fine-tuning of LLMs, federated learning and reinforcement learning. Finally, we discuss some promising directions and open problems for further research.