Exploring Approaches to Detection of Anomalies in Streaming Data

Numerous methods have been proposed to detect anomalies in data streams. In this work, a comprehensive study of performance of AutoEncoders and Predictive networks, applied to two datasets, is presented. The first dataset (from the paper mill) is labeled, whereas the second (from the server farm) is not. In this context, first, AutoEncoders and Predictive networks are applied to the labeled dataset and tuned, to improve their performance. Moreover, chronological and random training data splitting is explored. Additionally, an industry expert’s suggested performance evaluation method is proposed. Effects of its use are experimentally investigated. It is shown that the proposed approach outperforms the state-of-the-art approaches. The best of breed model and approach from the labeled paper mill dataset, is applied to the log data from a server farm. Obtained results turned out to “make sense” to the log data owners, and the developed method is going to be tried in a real-life deployment.