Scholarlink

With the evolution of the Internet of Things (IoT), a vast amount of data is generated from various intelligent devices and sensors. Data streaming in IoT systems may face quality issues like incompleteness due to sensor failures, network disruptions, or transmission errors. Addressing this problem is crucial, as unhandled missing data leads to inaccurate and unreliable analysis, compromising decision-making processes. This paper presents a multi-layer, data-driven methodology for imputing univariate IoT data streams with significant missing gaps by combining statistical analysis, signal processing, and Deep Learning algorithms. We propose a novel approach called the ES-CC-BiLSTM model, which consists of four phases: first, inserting missing data using the Exponential Smoothing technique; second, generating diverse statistical and temporal features; third, conducting cross-correlation analysis to identify the most critical features; and finally, utilizing the Bidirectional Long Short-Term Memory (BiLSTM) model to impute missing data. The model is evaluated using real-world water consumption datasets with artificially simulated missing values ranging from 10 to 50% and long interval gaps of up to 48 consecutive missing values. Experimental results demonstrate that the proposed framework reduces imputation error by up to 30% compared to the second best-performing model, and significantly outperforms other statistical and Machine Learning methods in terms of RMSE and MAE metrics.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A Novel Deep Learning-Based Multi-Layer Framework for Imputing IoT Data with Large Missing Gaps

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Hakob Grigoryan,
  • Dimitrios Gunopulos

摘要

With the evolution of the Internet of Things (IoT), a vast amount of data is generated from various intelligent devices and sensors. Data streaming in IoT systems may face quality issues like incompleteness due to sensor failures, network disruptions, or transmission errors. Addressing this problem is crucial, as unhandled missing data leads to inaccurate and unreliable analysis, compromising decision-making processes. This paper presents a multi-layer, data-driven methodology for imputing univariate IoT data streams with significant missing gaps by combining statistical analysis, signal processing, and Deep Learning algorithms. We propose a novel approach called the ES-CC-BiLSTM model, which consists of four phases: first, inserting missing data using the Exponential Smoothing technique; second, generating diverse statistical and temporal features; third, conducting cross-correlation analysis to identify the most critical features; and finally, utilizing the Bidirectional Long Short-Term Memory (BiLSTM) model to impute missing data. The model is evaluated using real-world water consumption datasets with artificially simulated missing values ranging from 10 to 50% and long interval gaps of up to 48 consecutive missing values. Experimental results demonstrate that the proposed framework reduces imputation error by up to 30% compared to the second best-performing model, and significantly outperforms other statistical and Machine Learning methods in terms of RMSE and MAE metrics.