The rapid advancement of machine learning (ML) technologies has garnered significant attention in various fields, including education. Predicting student performance is crucial for enhancing educational outcomes and identifying students at risk of underperforming. This systematic review evaluates the effectiveness of ML algorithms in predicting student academic performance in higher education. A comprehensive analysis of 50 peer-reviewed articles published between 2018 and 2023 was conducted, consolidating insights from 54 references. This review focuses on the accuracy, interpretability, scalability, and effectiveness of various ML algorithms used in student performance prediction. Results indicate that ensemble learning methods and deep learning models generally outperform traditional statistical techniques, providing higher accuracy in predicting student outcomes. However, their effectiveness is contingent upon the quality of input data, the granularity of features, and specific educational contexts. A comparative evaluation of accuracy, interpretability, and scalability of models is included. The paper also addresses the limitations, advantages, and drawbacks of these techniques, and highlights the need for further research in data privacy and ethical considerations.