Missing data is a problem commonly seen in most if not all real-world applications. Particularly for water quality monitoring systems, which are commonly plagued by sensor faults or network errors, missing or erroneous data pose a significant challenge in extracting accurate and meaningful insights. In this work, we investigate the problem of missing information in time-series data and propose a new method DISC - Data Imputation with Seasonality and Causality - which uses the concepts of seasonal decomposition and causal discovery to improve contextual accuracy of the imputations for time-series. DISC operates in two stages. First, it builds a causal relational graph representing inter-feature dependencies and uses this graph to impute missing values by adjusting estimates to the nearest-neighbour hourly data points. Second, it learns yearly, monthly, and daily seasonal patterns at an hourly resolution and imputes the remaining gaps. In scenarios where seasonal decomposition fails to fully resolve gaps, causal discovery exploits dependencies among time-series features to generate reference points that enhance the completeness of the seasonal pattern. The proposed method has been evaluated using real-world data from the Murray-Darling Basin and compared with multiple existing machine learning methods. The results validate the effectiveness of DISC, enabling accurate imputation of 14 consecutive days of missing hourly data with an R-squared of 80%, more than twice the performance of conventional approaches.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Effective Missing-Data Imputation for Time Series with Seasonality and Causality

  • Devesh Bhogal,
  • Yanchang Zhao,
  • Reena Kapoor,
  • Tapas Biswas,
  • Peter Toscas,
  • Klaus Joehnk

摘要

Missing data is a problem commonly seen in most if not all real-world applications. Particularly for water quality monitoring systems, which are commonly plagued by sensor faults or network errors, missing or erroneous data pose a significant challenge in extracting accurate and meaningful insights. In this work, we investigate the problem of missing information in time-series data and propose a new method DISC - Data Imputation with Seasonality and Causality - which uses the concepts of seasonal decomposition and causal discovery to improve contextual accuracy of the imputations for time-series. DISC operates in two stages. First, it builds a causal relational graph representing inter-feature dependencies and uses this graph to impute missing values by adjusting estimates to the nearest-neighbour hourly data points. Second, it learns yearly, monthly, and daily seasonal patterns at an hourly resolution and imputes the remaining gaps. In scenarios where seasonal decomposition fails to fully resolve gaps, causal discovery exploits dependencies among time-series features to generate reference points that enhance the completeness of the seasonal pattern. The proposed method has been evaluated using real-world data from the Murray-Darling Basin and compared with multiple existing machine learning methods. The results validate the effectiveness of DISC, enabling accurate imputation of 14 consecutive days of missing hourly data with an R-squared of 80%, more than twice the performance of conventional approaches.