Cybersecurity has become essential worldwide and at all levels, concerning individuals, institutions, and governments. A basic principle in cybersecurity is to be always alert. Therefore, automation is imperative in processes where the volume of daily operations is large. Several cybersecurity applications can be addressed as binary classification problems, including anomaly detection, fraud detection, intrusion detection, spam detection, or malware detection. In many cases, the positive class samples, those that represent a problem, occur at a much lower frequency than negative samples, and this poses a challenge for machine learning algorithms since learning patterns out of under-represented samples is hard. This is known in machine learning as imbalance learning. In this study, we systematically evaluate various machine learning methods using two representative financial datasets containing numerical and categorical features. The Credit Card dataset contains 283726 samples, 31 features, and 0.2 percent of the transactions are fraudulent (imbalance ratio of 598.84:1). The PaySim dataset contains 6362620 samples, 11 features and 0.13 percent of the transactions are fraudulent (imbalance ratio of 773.70:1). We present three experiments. In the first experiment, we evaluate single classifiers including Random Forests, Light Gradient Boosting Machine, eXtreme Gradient Boosting, Logistic Regression, Decision Tree, and Gradient Boosting Decision Tree. In the second experiment, we test different sampling techniques including over-sampling, under-sampling, Synthetic Minority Over-sampling Techique, and Self-Paced Ensembling. In the last experiment, we evaluate Self-Paced Ensembling and its number of base classifiers. We found that imbalance learning techniques had positive and negative effects, as reported in related studies. Thus, these techniques should be applied with caution. Besides, we found different best performers for each dataset. Therefore, we recommend testing single classifiers and imbalance learning techniques for each new dataset and application involving imbalanced datasets as is the case in several cyber security applications. We provide the code with all experiments as open-source (Available at https://github.com/MateoLopez00/Imbalanced-Learning-Empirical-Evaluation .).