Log parsing is the basis of log anomaly detection in security operations, which converts raw log messages into structured data. The existing log parsing is only a binary classification task, categorizing log message tokens as either constant or variable without considering the underlying semantic context. In contrast, log semantic parsing represents a more advanced technique, operating as a multi-classification task that assigns each token in a log message to either a constant or its corresponding variable type. It enables a deeper comprehension of the semantic nuances within log data, thereby facilitating more insightful log mining processes and analyses. However, the current public data sets only include variable or constant labels, and lack of multiple variable types with semantics. The existing semantic log parser (Semparser) not only has low parsing efficiency in training and parsing, but also requires high server configuration. To address the limitations of existing methods, in this paper, we propose Log Semantic Parsing Based on Sequence Annotation (LogSPSA). LogSPSA utilizes a novel sequence annotation method to identify constants and corresponding variable types based on few labeled semantic log data. In addition, an adaptive random sampling algorithm is designed to select a small and diverse training set. Furthermore, we construct corresponding virtual vectors for all variable types in the small sample set. The classification task for each token is converted to the sequence annotation task for each log message. We conduct extensive experiments on seven public log datasets. The experimental results show that LogSPSA is effective and efficient for log semantic parsing.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Log Semantic Parsing Based on Sequence Annotation in Security Operations

  • Shiming He,
  • Siru Long,
  • Gan Wu,
  • Diqing Liang,
  • Kun Xie

摘要

Log parsing is the basis of log anomaly detection in security operations, which converts raw log messages into structured data. The existing log parsing is only a binary classification task, categorizing log message tokens as either constant or variable without considering the underlying semantic context. In contrast, log semantic parsing represents a more advanced technique, operating as a multi-classification task that assigns each token in a log message to either a constant or its corresponding variable type. It enables a deeper comprehension of the semantic nuances within log data, thereby facilitating more insightful log mining processes and analyses. However, the current public data sets only include variable or constant labels, and lack of multiple variable types with semantics. The existing semantic log parser (Semparser) not only has low parsing efficiency in training and parsing, but also requires high server configuration. To address the limitations of existing methods, in this paper, we propose Log Semantic Parsing Based on Sequence Annotation (LogSPSA). LogSPSA utilizes a novel sequence annotation method to identify constants and corresponding variable types based on few labeled semantic log data. In addition, an adaptive random sampling algorithm is designed to select a small and diverse training set. Furthermore, we construct corresponding virtual vectors for all variable types in the small sample set. The classification task for each token is converted to the sequence annotation task for each log message. We conduct extensive experiments on seven public log datasets. The experimental results show that LogSPSA is effective and efficient for log semantic parsing.