This paper presents a comparative analysis of different machine learning models and calibration techniques for disease diagnosis using calibrated probability estimates. The paper uses two datasets, one simulated and one real, with similar degrees of class imbalance, to evaluate the performance of six classifiers: logistic regression, Naive Bayes, linear discriminant analysis, support vector machine, decision tree, and K-nearest neighbor. The paper also introduces two novel calibration methods, beta calibration and spline calibration, and compares them with the existing methods of Sigmoid calibration and isotonic regression. Various evaluation metrics were employed, such as ROC-AUC, F1-score, Jaccard score, stratified Brier score, log loss, accuracy, and MCC, to assess the accuracy and reliability of the calibrated predictions. Results from this research work showed that spline calibration performs well across all classifiers and datasets, improving the calibration and discrimination of the predicted probabilities. The paper also discusses the implications and limitations of the proposed methods and suggests future research directions.