Improving Missing Data Imputation with GF-WAI: An Explainable AI-Based Ensemble Method for Dengue Disease Prediction
摘要
Missing data has been a significant challenge in data analysis, reducing the reliability of predictions and disrupting data patterns. In dengue-related datasets, incomplete data has complicated outbreak prediction and public health responses. Traditional imputation techniques often produce biased results, while advanced techniques are computationally intensive and lack transparency. To address this, a novel ensemble weighted average imputation technique combining XGBoost and MiceForest named Gradient-Forest Weighted Average Imputer (GF-WAI) has been proposed. Six additional imputation techniques have been implemented for comparison, and predictive accuracy has been evaluated using RFC, SVC, and NBC. Evaluations have been conducted on both dengue datasets of 1,003 and 10,000 records. The proposed method has outperformed others, achieving an MAE of 641.26 and RMSE of 6743.63 on the small dataset, with significant improvements on the larger dataset, where an MAE of 144.49 and RMSE of 2408.32 have been achieved. An accuracy of 99.90% and an F1 score of 99.89% have been achieved by GF-WAI with RFC on the larger dataset, while 98.26% accuracy and an F1 score of 98.25% have been achieved on the smaller dataset. LIME, an explainable AI tool, has been used to further evaluate GF-WAI, providing insights into feature contributions. These findings have highlighted the effectiveness of the proposed technique in addressing missing data challenges, supporting better public health decision-making for dengue intervention strategies.