<p>Missing data are a pervasive challenge in data-intensive science and engineering, where imputation must recover incomplete observations while preserving downstream predictive performance. Ultra-data-oriented parallel fractional hot-deck imputation (UP-FHDI) is an assumption-free and scalable approach for large, high-dimensional incomplete data. However, its practical value and configuration sensitivity cannot be fully characterized by imputation accuracy alone. To address this gap, we propose an imputation-and-prediction integrated framework (IPIF) that jointly evaluates imputation methods from two complementary perspectives: imputation quality and impact on downstream predictive tasks. Unlike conventional approaches that treat imputation as an isolated preprocessing step, IPIF provides a unified and modular pipeline in which imputation methods and predictive models can be flexibly integrated. To further enhance scalability, we refine the parallel workflow of UP-FHDI through optimized file system and communication strategies, significantly reducing I/O overhead in large-scale settings. Extensive experiments on synthetic and real-world datasets demonstrate that UP-FHDI consistently outperforms state-of-the-art baselines in imputation accuracy and yields improved downstream predictive performance. Notably, the results reveal that lower imputation error does not necessarily translate into better predictive outcomes, underscoring the importance of task-oriented evaluation. Finally, sensitivity analyses of the UP-FHDI configurations within IPIF indicate that selecting 70–90 important variables achieves a favorable balance between imputation effectiveness and predictive performance.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

IPIF: an imputation-and-prediction integrated framework for ultra-data-oriented parallel fractional hot-deck imputation

  • Yicheng Yang,
  • Zixian Li,
  • Tianyi Li,
  • Yongfeng Dong,
  • Yanyan Huang

摘要

Missing data are a pervasive challenge in data-intensive science and engineering, where imputation must recover incomplete observations while preserving downstream predictive performance. Ultra-data-oriented parallel fractional hot-deck imputation (UP-FHDI) is an assumption-free and scalable approach for large, high-dimensional incomplete data. However, its practical value and configuration sensitivity cannot be fully characterized by imputation accuracy alone. To address this gap, we propose an imputation-and-prediction integrated framework (IPIF) that jointly evaluates imputation methods from two complementary perspectives: imputation quality and impact on downstream predictive tasks. Unlike conventional approaches that treat imputation as an isolated preprocessing step, IPIF provides a unified and modular pipeline in which imputation methods and predictive models can be flexibly integrated. To further enhance scalability, we refine the parallel workflow of UP-FHDI through optimized file system and communication strategies, significantly reducing I/O overhead in large-scale settings. Extensive experiments on synthetic and real-world datasets demonstrate that UP-FHDI consistently outperforms state-of-the-art baselines in imputation accuracy and yields improved downstream predictive performance. Notably, the results reveal that lower imputation error does not necessarily translate into better predictive outcomes, underscoring the importance of task-oriented evaluation. Finally, sensitivity analyses of the UP-FHDI configurations within IPIF indicate that selecting 70–90 important variables achieves a favorable balance between imputation effectiveness and predictive performance.