This paper aims to comprehensively assess the performance evaluation metrics Matthews correlation coefficient (MCC), F1 score, and balanced accuracy when applied to machine learning models dealing with imbalanced health datasets. Given the challenges posed by uneven class distributions in health data, understanding the behavior of these metrics becomes crucial for accurate model assessment. The Matthews correlation coefficient, renowned for its ability to account for true positives, true negatives, false positives, and false negatives, offers a holistic view of classification performance, especially vital in imbalanced scenarios. Contrarily, the F1 score, which harmonizes precision and recall, emphasizes the significance of managing false predictions in health-related classifications, aiming to strike a balance between these measures. Meanwhile, balanced accuracy ensures a fair appraisal of model performance by considering sensitivity and specificity, mitigating biases that stem from class imbalances. By subjecting these metrics to rigorous experimental analyses using diverse imbalanced health datasets, this research seeks to unravel their applicability, strengths, and limitations in evaluating machine learning models in the realm of health informatics. Ultimately, this investigation endeavors to refine assessment methodologies, contributing to the development of more robust, dependable predictive models tailored for healthcare applications.