Log Parsing with LLMs Featuring Self-reflection and Continuous Refining
摘要
Log parsing is a critical step in analyzing logs from large-scale software systems, enabling tasks like anomaly detection and network failure monitoring. Existing approaches, including statistical and deep learning-based methods, often lack semantic understanding or struggle with unseen data. Recently, large language models (LLMs) have shown promise in high-precision log parsing but face two major challenges: misidentification of constants and variables, and narrow-scope processing that overlooks contextual relationships across logs. To address these challenges, we propose LogReflex, an online log parsing framework leveraging LLMs. LogReflex introduces a self-reflection mechanism to correct misidentified constants and variables within extracted templates and a template refining mechanism to incorporate historical templates into the LLM query, enhancing contextual understanding while maintaining online parsing capabilities. The framework integrates five key components, including lightweight modules to improve accuracy with minimal system overhead. Extensive experiments on benchmark datasets demonstrate that LogReflex outperforms state-of-the-art approaches.