A Dirty Data Detection Method for Stream Data Cleaning of Online Machinery Condition Monitoring

Dirty data are commonly seen during online machinery condition monitoring and lead to the decrease of stream data quality. By analyzing these poor-quality data, unreliable or misleading results are probably obtained. To improve the online monitoring stream data quality, a dirty data detection method based on incremental local outlier factor is proposed. The proposed method consists of two stages. In the first stage, historical data are used to calculate local outlier factor of these data, which is performed off-line. In the second stage, the local outlier factor of the new coming stream data is calculated, and local outlier factor of historical data are updated dynamically when new coming data collected from online monitoring system is inserted or dirty data is deleted. Then, the new coming data whose local outlier factor is larger than the predefined threshold value are carefully judged to determine whether the data are dirty data or trend data. One real case concerning a roller press of raw grinding mill used for cement production is applied to verify the effectiveness of the proposed method. The results demonstrate that the proposed method is able to detect dirty data but do less harm to timeliness and thus has broad application prospect in online machinery condition monitoring system for dirty data cleaning.