Fairness-preserving framework for machine learning: data bias quantification, model evaluation, and robustness across multiple datasets
摘要
Fairness in machine learning systems is crucial for developing trustworthy, ethical, and socially responsible AI, particularly in high-stakes domains such as healthcare and social services. This study proposes a comprehensive fairness-preserving framework that integrates data bias quantification with model-level fairness evaluation, thereby eliminating its violation. The framework uses Earth Mover’s Distance to quantify the distributional discrepancy between subgroups and the overall population, providing a statistical foundation for identifying group-level data bias. We implement the framework using two real-world health datasets and evaluate fairness across eight widely accepted definitions: demographic parity, equalised odds, equal opportunity, false positive rate parity, treatment equality, discrimination score, disparate impact, and balanced error rate. Five commonly used supervised learning algorithms (Logistic Regression, Decision Tree, Random Forest, Support Vector Machine, and K-Nearest Neighbours) are used to assess model performance and fairness outcomes. Results demonstrate that fairness-preserving adjustments, primarily through targeted data modification, significantly reduce fairness violations with minimal impact on predictive accuracy. To further assess robustness, we conducted an ablation study using a third dataset, comparing three implementation strategies: conventional machine learning, fairness-aware learning via the Fairlearn toolkit, and our proposed approach. While Fairlearn reduced fairness violations from 25 to 18 (out of 40), our approach eliminated them entirely. These results highlight the limitations of model-level mitigation and demonstrate the effectiveness of data-level bias removal in achieving fully fair outcomes. By combining data-level bias quantification with robust statistical validation, this work offers a practical and interpretable approach to implementing fairness in ML systems, contributing to equitable and accountable AI.