The use of digital payment systems has transformed the process of financial transactions, with ease and efficiency in providing payment services in most parts of the world. Nevertheless, this development has increased cybersecurity threats where fraudulent transactions becoming a major threat. Such scams are associated with monetary losses and loss of trust to online services, and this is why effective detection mechanisms must be established. In this paper, the author suggests the machine learning (ML)-based framework of fraudulent transactions identification in the online payment systems. There are seven models that were tested, which include Logistic regression (LR), K-Nearest Neighbor (KNN), Decision Trees (DT), Random Forest (RF), Extreme Gradient Boosting (XGB), Naive Bayes (NB) and Support Vector Classifier (SVC). In order to overcome the issue of unbalanced data, oversampling and undersampling were performed and model sensitivity to minority-class samples increased. It was found from the results of the experiments that ensemble models, XGB, and RF provided the best outcomes in terms of accuracy with the values of 99.3% and 99.4%, respectively. These models produced higher results than other simpler models like LR and NB on important parameters like precision, recalls, and F1-score. This study points out how ML can enhance cybersecurity in online payment services. Such issues as data imbalance and scalability can be overcome by the suggested framework, which can be of service in creating the next-generation fraud detection systems, raising the level of trust and security of the digital economy.