TWLog: Task Workflow-Based Log Anomaly Detection

Log anomaly detection is crucial for pinpointing software issues, particularly in large-scale distributed systems where numerous services span multiple machines. Logs are abundant in such environment, meanwhile the log sequences lack inherent business logic when examined solely based on timestamps. Moreover, the parallel execution of tasks leads to interleaved logs which caused log entries belonging to same task span a wide range of timestamps. Existing anomaly detection approaches ignore the complex structure of a trace brought by its invocation hierarchy. In addition, most of them based on RNNs which is difficulty in capturing long-range dependencies and the challenge of handling variable-length sequences effectively. In this paper, we propose TWLog, a self-supervised deep learning-based method. TWLog uses a unified trace graph to describe the complex structure of a trace combining the task workflow and log events. Based on the basic task workflow from log message, we extract the semantic information from raw log messages as vector representations. These vectors are then fed into a Transformer-based model which can capture the contextual information from task workflow-based log sequences. The experiments on real-world dataset of HDFS, OpenStack and Kubernetes confirm the effectiveness of our method.