Probabilistic modeling is a statistical approach that utilizes the outcome of random events to predict the possibility of future results, and a sound knowledge of it aids decision-making among several alternatives. Unfortunately, traditional statistics use of P-values and misunderstanding of machine learning (ML) probabilistic approach have deterred the interpretation of many projects which have adversely affected many research findings. Hence, the objective of this study is to clarify the place of probability and machine learning algorithms’ performance metrics in biomedical informatics of heart disease. Logistic Regression (LR), Linear Discriminant Analysis (LDA), and Quadratic Discriminant Analysis (QDA) were adopted to model the prediction of heart disease with the dataset from the Kaggle repository. The findings revealed that although ROC-AUC pointed out that QDA outperformed LR and LDA with AUC; 0.916, LR; 0.902, LDA; 0.906, the False Positive Rate (FPR) and True Positive Rate (TPR) of the AUC could not reveal any threshold (probability) values associated with the FPR and TPR of the test observations. As a result, we used Python to identify the FPR, TPR values, and their corresponding thresholds. The findings revealed that the probability values are within the thresholds and showed that the probability 0.188478 maximized the TPR (sensitivity) performance with 0.887850 (89%). The findings also revealed an inverse relationship between TPR and its threshold. The study further clarified the interpretation/implications of the confusion matrix to probability. It also demonstrated how the odds ratio could interpret the impacts of the predictors: Chest Pain Type, ST_Slope, Old-peak, and increase in the exercise-induced ST depression on heart disease against the traditional statistics P-values.