Log anomaly detection based on BERT

With the increasing complexity of computing clusters and large-scale network systems, anomaly detection based on logs has gained significant attention to identify system issues caused by machine failures or malicious attacks. To capture contextual information and local features in log sequences effectively, BERT (Bidirectional Encoder Representation from Transformers) with separated score attention and dual-branch (SD-BERT), a log anomaly detection method derived from BERT encoder blocks is introduced. SD-BERT employs normal log sequences as the training data and is trained by predicting masked log keys. In SD-BERT, taking into account the characteristics of log anomaly detection tasks, we redesign the scoring mechanism and propose the separated score attention (SSA). This helps enhance the model's attention towards different tokens and positions in a sequence. Since log sequence anomalies are related to partial segments in the sequence, a dual-branch module is designed with an SSA branch and a convolutional branch. The SSA branch is capable of capturing the global context related to the abnormal position, while the convolutional branch helps capture local abnormal details. This dual-branch design enables the model to have a more comprehensive understanding and detection of anomalous behavior in log sequences. A series of comparative experiments are conducted on HDFS, BGL, and Thunderbird datasets. The experimental results demonstrate that SD-BERT exhibits comparable or superior performance in contrast to the compared models, confirming the superiority of SD-BERT in log anomaly detection.