Machine Learning and Anomaly Detection for an Automated Monitoring of Log Data

Log data provide an important source of information to understand the internal processes of software systems, and they are a valuable source to analyze the status of a software system. However, the large number of log entries makes manual analysis of the log data difficult. In order to identify errors or disruptions of computer systems at an early stage, an automated evaluation of the log data using artificial intelligence can be useful. In this work, three common methods ranging from conventional machine learning methods, i.e., clustering the log data, to deep learning are examined to monitor log data in real time and thus identify anomalous behavior of computer systems at an early stage. We show that the success of an algorithm is dependent on the form of the log entries as well as the preprocessing steps performed on the log entries. This way, we apply algorithms on common semi-structured log data, exception fingerprints, and well-structured authentication logs. While it is clear that different data sets might require different models in order to identify an anomaly and thus a possible attack, clustering methods prove to be efficient and accurate in most cases.