Bayesian optimization (BO) has emerged as a popular approach for optimizing expensive black-box functions, which are common in modern machine learning, scientific research, and industrial design. This paper provides a comprehensive review of the recent advances in Bayesian optimization techniques, addressing new methodological developments such as multi-fidelity optimization, transfer learning, and neural network surrogates. Additionally, we explore the increasing role of BO in complex, high-dimensional, and multi-objective optimization problems, as well as its application in various fields like hyperparameter tuning, reinforcement learning, and neural architecture search. The goal of this review is to offer both theoretical insights and practical guidelines to researchers and practitioners working in areas where BO is a suitable tool. Finally, we discuss key challenges and propose directions for future research in the rapidly evolving field of Bayesian optimization.