ShellMamba: Anomaly Detection of Shell Command Sequences

Shell command anomaly detection plays a vital role in securing large-scale information systems, where malicious or erroneous commands may introduce significant operational risks. However, existing methods often fail to jointly model fine-grained command semantics and long-range contextual dependencies, limiting their ability to detect subtle anomalies. To address this issue, we propose ShellMamba, a unified framework for representation learning and anomaly detection over shell command sequences. ShellMamba employs a multi-granularity learning strategy that captures both semantic structures and temporal dependencies through masked sequence modeling and contextual prediction. Furthermore, we design a two-stage detection mechanism that combines session-level reconstruction with command-level predictive modeling. This design enables the identification of both global behavioral deviations and fine-grained anomalous commands. Experimental results on real-world enterprise datasets show that ShellMamba consistently outperforms existing methods, achieving average improvements of 6.5% in precision, 10.5% in recall, and 8.7% in F1-score. Notably, the proposed self-supervised framework eliminates the need for labeled anomalies while maintaining effective optimization via multi-scale consistency learning.